USDA 



United States Department of Agriculture 


Community Eligibility Provision Evaluation 


Nutrition Assistance Program Report February 2014 

Food and Nutrition Service 
Office of Policy Support 




USDA is an Equal Opportunity Provider and Employer 



USDA 



United States Department of Agriculture 


Food and Nutrition Service, Office of Policy Support 


February 2014 


Community Eligibility Provision Evaluation 


Authors: 

Christopher W. Logan 

Patty Connor 

Eleanor L. Harvill 

Joseph Harkness 

Hiren Nisar 

Amy Checkoway 

Laura R. Peck 

Azim Shivji 

Edwin Bein 

Marjorie Levin 

Ayesha Enver 


Submitted by: 

Submitted to: 

Abt Associates 

Office of Policy Support 

55 Wheeler St. 

Food and Nutrition Service 

Cambridge, MA 02138 

3101 Park Center Drive 
Alexandria, VA 22302 

Project Director: 

Project Officer: 

Patty Connor 

John R. Endahl 


This study was conducted under Contract number AG-3198-D-l 1-0074 with the Food and 
Nutrition Service, United States Department of Agriculture. 

This report is available on the Food and Nutrition website: 
http ://www. fns .usda. gov / research-and-analysis 

Suggested Citation: 

Logan, Christopher W., Patty Connor, Eleanor L. Harvill, Joseph Harkness, Hiren Nisar, 
Amy Checkoway, Laura R. Peck, Azim Shivji, Edwin Bein, Marjorie Levin, and Ayesha 
Enver. Community Eligibility Provision Evaluation. Project Officer: John R. Endahl. 
Prepared by Abt Associates for the U.S. Department of Agriculture, Food and Nutrition 
Service, February 2014. 








Community Eligibility Provision Evaluation: Final Report 


Acknowledgements 

The Community Eligibility Provision (CEP) Evaluation was conducted under the direction of the 
Office of Policy Support, Food and Nutrition Service (FNS), U.S. Department of Agriculture. We 
would like to tha nk our FNS Project Officer, John Endahl and his colleague at FNS, Margaret 
Applebaum for their guidance and support throughout the project and for their valuable insights and 
feedback on this report specifically. 

The authors also want to acknowledge the valuable contributions of other Abt Associates team 
members who worked on instrument development, recruiting FEAs, data collection, data processing 
and analysis of the data, including Meryl Finkel, Michele Mendelson, with assistance from Brenda 
Rodriguez and Brianna Roche from Abt-SRBI, Vinh Tran, Marc Moss, Tara Wommack, Nancy 
McGarry, Samantha Crowell, Kelly Patlan, Cora Goldston, RJ DelaCruz, Don FaFiberty, and our 
school nutrition consultants, Mary Jo Cutler and Tami Cline. Other Abt project staff contributed by 
conducting or assisting with the interviews of State Child Nutrition Officials and State Title I 
Directors in one or both rounds of interviews: Marjorie Fevin, Porsha Cropper, Evan Easterbrooks- 
Dick, Sean Morris, and Rebecca Gotlieb. 

We also want to tha nk Susan Bartlett, Beth Gamse and Carter Epstein who provided valuable 
comments on drafts of this report. Patricia McKinney, our expert consultant, also provided helpful 
insight and co mm ents as the technical reviewer. Tami Cline of Cline Consulting provided valuable 
comments and school foodservice perpective on the final report draft. Maria Boyle and Fauren Olsho 
also provided valuable comments to the evaluation team. We would also like to thank Jan Nicholson, 
our intrepid production specialist. 

Finally, the authors thank the State Child Nutrition staff in all participating Community Eligibility 
Provision States for their valuable time spent providing the information that contributed to this report, 
through interviews and data files. In addition, the evaluation team would like to thank the school 
foodservice and FEA staff that participated in the study by completing, surveys, interviews and 
facilitating the on-site data collection in their schools and FEA offices. We are grateful for the time 
the time they devoted to providing us with the data necessary to conduct this important evaluation. 


Abt Associates 


Acknowledgements | pg. i 





Community Eligibility Provision Evaluation: Final Report 


List of Acronyms and Abbreviations 


ADP 

CCD 

CEO 

CEP 

CITS 

CN 

DCM 

ELL 

EN 

EP 

ESEA 

FDPIR 

FNS 

FPL 

FRP 

FSD 

FSMC 

FY 

HHFKA 

IOM 

ISP 

LEA 

MANOVA 

NCES 

NE 

NSLA 

NSLP 

NSMP 

ovs 

PEAR survey 

POS 

PPS 

PSM 

PSUS 

SBP 

SD 

SE 

SEA 

SFA 

SNAP 

SNDA 

SY 

TANF 
USD A 
VSR 


Average daily participation 

National Center for Educational Statistics’ Common Core of Data 
Community Eligibility Option now known as CEP (see below) 
Community Eligibility Provision 
Comparative interrupted time series 
Child Nutrition 

Direct Certification with Medicaid 
English Language Learners 
Eligible Non-Participating 
Eligible Participating 

Elementary and Secondary Education Act of 1965 

Food Distribution Program on Indian Reservations 

Food and Nutrition Service 

Federal poverty line 

Free or reduced price 

Foodservice director 

Foodservice management company 

Fiscal Year 

Healthy, Hunger Free Kids Act of 2010 
Institute of Medicine 
Identified student percentage 
Local Education Agency 
Multivariate analysis of variance 
National Center for Educational Statistics 
Near-Eligible 

National School Lunch Act 
National School Lunch Program 
Nutrient Standard Menu Planning 
Offer vs. Serve 

Participation, enrollment, attendance and revenue survey 
Point of Sale 

Probability proportional to size 

Propensity score matching 

Private School Universe Survey 

School Breakfast Program 

Standard deviation 

Standard error 

State Education Agency 

School Food Authority 

Supplemental Nutrition Assistance Program 

School Nutrition Dietary Assessment Study 

School Year 

Temporary Assistance for Needy Families 
United States Department of Agriculture 
FNS Verification Summary Report (FNS-742) 


Abt Associates 


List of Acronyms and Abbreviations | pg. iii 





Community Eligibility Provision Evaluation: Final Report 


Table of Contents 

Executive Summary.1 

Study Design and Data.2 

Implementation Study Results.2 

Impact Study Results.4 

Summary.6 

1. Introduction.9 

1.1 Study Background.10 

1.1.1 Authorization for the CEP and Mandate for the Evaluation.11 

1.1.2 Purpose and Design of the CEP.11 

1.2 Study Objectives and Overview.12 

1.2.1 Implementation Study Objectives.13 

1.2.2 Impact Study Objectives and Research Questions.13 

1.3 Overview of Study Samples.15 

1.3.1 Administrative Data for the Universe of LEAs.15 

1.3.2 Implementation Study LEA Survey Sample.15 

1.3.3 Matched Administrative Data Sample for the Participation and Revenue 

Analysis.17 

1.3.4 Component 1: Participation, Enrollment, Attendance and Revenue 

(PEAR) Survey.17 

1.3.5 Component 2: Administrative Costs and Certification Errors Data.17 

1.3.6 Component 3: Meal Quality and Meal Counting/Claiming Data.17 

1.4 Scope of the Report.17 

2. Study Design, Sampling, and Data.19 

2.1 Implementation Study Design, Sampling, and Data.19 

2.1.1 Design and Sampling.19 

2.1.2 Administrative and Extant Data.21 

2.1.3 LEA Web Surveys for Implementation Study.21 

2.1.4 State Child Nutrition Director Interviews.23 

2.2 Impact Study Design.24 

2.3 Impact Study Sampling.27 

2.3.1 Target Population.28 

2.3.2 Matched Administrative Data Sample: Procedure and Baseline 

Characteristics.29 

2.3.3 Primary Data Collection Sample: Matching Procedure and Baseline 

Characteristics.31 

2.4 Impact Study Data Collection.34 

2.4.1 Administrative Data on Participation, Enrollment, Attendance and 

Revenue.38 

2.4.2 Participation, Enrollment, Attendance and Revenue (PEAR) Survey.38 

2.4.3 Administrative Cost Interviews.39 

2.4.4 Certification Record Reviews.40 

2.4.5 Menu Survey.40 


Abt Associates 


Contents | pg. v 









































Community Eligibility Provision Evaluation: Final Report 

2.4.6 Cashier Observations.40 

2.4.7 Meal Counting and Claiming.41 

3. Implementation of the Community Eligibility Provision.43 

3.1 Decision-Making and Notification of LEAs, Schools and the Community.43 

3.1.1 State Participation Decision.44 

3.1.2 LEA Participation Decision Process.44 

3.1.3 State Communications with LEAs.45 

3.1.4 LEA Communications with Schools and Community Members.48 

3.2 Benefits and Barriers to CEP Participation.50 

3.2.1 Perceived Benefits.50 

3.2.2 Perceived Barriers.55 

3.2.3 State Perceptions of Types of LEAs Most Likely to Participate.57 

3.3 State-Level and LEA-Level Implementation.58 

3.3.1 Adaptation of Meal Reimbursement Systems.58 

3.3.2 Direct Certification Process.59 

3.3.3 Changes to Meal Variety, Staffing, Process & Cost.60 

3.4 Results and Lessons Learned from CEP Implementation.60 

3.4.1 Benefits and Challenges for State and Local Implementation of the CEP.60 

3.4.2 Benefits and Barriers to LEA Participation in the CEP.61 

3.4.3 Impacts of the CEP on Allocation of Educational Funds.62 

3.4.4 Summary of Implementation Results.64 

4. Take-Up of the Community Eligibility Provision.65 

4.1 Numbers of Near-Eligible, Eligible Non-Participating, and Participating LEAs.65 

4.1.1 Regular LEAs: Numbers of Near-Eligible, Eligible Non-Participating, 

and Participating.65 

4.1.2 Other LEAs: Numbers of Near-Eligible, Eligible Non-Participating, and 

Participating.66 

4.1.3 Eligible School Participation in CEP LEAs.67 

4.1.4 LEA Approaches to Claiming Percentages.68 

4.2 Characteristics of Near-Eligible, Eligible Non-Participating, and Participating 

LEAs and Schools.69 

4.2.1 Overall Patterns of the Characteristics of Near-Eligible, Eligible Non- 

Participating, and Participating LEAs.69 

4.2.2 State Variations in the Characteristics of Near-Eligible, Eligible Non- 

Participating, and Participating LEAs.75 

4.2.3 Characteristics of Near-Eligible, Eligible Non-Participating, and 

Participating Schools.79 

4.2.4 Key Results of Descriptive Analysis.83 

4.3 Factors Associated with Take-Up of the CEP.84 

4.3.1 Model Specification.84 

4.3.2 Results.86 

4.4 Summary.88 


pg. vi | Contents 


Abt Associates 








































Community Eligibility Provision Evaluation: Final Report 


5. Participation and Revenue Impacts.91 

5.1 Expectations about CEP Impacts on School Meals Participation, Revenues, and 

SBP Availability and Operations.93 

5.2 Outcome Measures.94 

5.2.1 Participation.94 

5.2.2 Federal Revenue.94 

5.2.3 Non-Federal Revenues.95 

5.3 Constructing Matched Comparison Groups.97 

5.3.1 Comparison Group Matching for Analyses of Participation and Federal 

Revenues.97 

5.3.2 Comparison Group Matching for Analyses of Non-Federal Revenues.98 

5.4 Analytic Model.99 

5.5 Impact of CEP on Participation in School Meal Programs.99 

5.5.1 Discussion of Participation Results.103 

5.6 Impacts of CEP on LEA Revenues from the School Meal Programs.104 

5.6.1 Federal Revenues.104 

5.6.2 Non-Federal Revenue.107 

5.6.3 Discussion of Revenue Results.110 

5.7 Impact on the Availability of the School Breakfast Program and Types of School 

Breakfast Service.110 

5.7.1 Presence of the School Breakfast Program.Ill 

5.7.2 Type of Breakfast Service Offered.112 

5.7.3 Student Choice of Foods at Breakfast.114 

5.7.4 Time Allowed for Students to Eat Breakfast.115 

5.7.5 Discussion of School Breakfast Program Results.115 

6. CEP Impacts on Staffing, Administrative, and Meal Production Costs.117 

6.1 Expectations for CEP Impacts on Staffing, Administrative, and Meal Production 

Costs.118 

6.2 Outcome Measures.118 

6.3 Data and Methods.119 

6.3.1 Data.119 

6.3.2 Analytic Model.119 

6.4 CEP Impacts on Administrative Staff Time and Costs.120 

6.4.1 CEP Impacts on Administrative Staff Effort and Costs Overall and by 

Type of Activity.120 

6.4.2 CEP Impacts on Administrative Staff Effort and Costs by Type of Staff.123 

6.5 Descriptive Findings for CEP Impacts on Meal Production Costs.124 

7. Program Integrity Impacts.127 

7.1 Administrative Errors Associated with the Certification Process.128 

7.1.1 Expectations for CEP Impacts on Certification and Procedural Errors.129 

7.1.2 Outcome Measures.129 

7.1.3 Data and Methods.132 

7.1.4 Impacts on Schools’ Certification Error.133 

7.1.5 Impacts on Schools’ Procedural Errors.134 


Abt Associates 


Contents | pg. vii 











































Community Eligibility Provision Evaluation: Final Report 


7.2 Cashier Errors. 135 

7.2.1 Expectations about CEP Impacts on Cashier Errors. 135 

7.2.2 Outcome Measures.136 

7.2.3 Data and Final Analytic Sample. 136 

7.2.4 Analytic Model. 137 

7.2.5 Cashier Error Impact Results. 138 

7.2.6 Discussion of the Results.139 

7.3 LEA Claiming Errors. 139 

7.3.1 Expectations about CEP Impacts on Claiming Errors. 139 

7.3.2 Outcome Measures.140 

7.3.3 Data and Final Analytic Sample. 141 

7.3.4 Analytic Model. 142 

7.3.5 Claiming Error Results. 142 

7.3.6 Discussion of the Results.144 

8. Nutritional Quality Impacts.147 

8.1 Expectations about CEP Impacts on Meal Quality and Variety. 148 

8.1.1 Research Questions Related to Meal Quality and Variety. 149 

8.2 Outcome Measures. 149 

8.2.1 Measures Used for NSLP Meal Quality Assessment. 150 

8.2.2 Measures Used for SBP Meal Quality Assessment. 151 

8.2.3 Measures Used for the Variety of Food Choices for NSLP and SBP.152 

8.3 Data and Methods. 152 

8.4 Meal Quality and Choice Results. 153 

8.4.1 Meal Quality. 154 

8.4.2 Menu Choice and Variety. 158 

8.4.3 Discussion. 160 

9. Conclusions.163 

9.1 Implementation Study Results. 163 

9.2 Impact Study Results. 164 

References.169 

Glossary: Community Eligibility Provision Evaluation.171 

Appendices 

Note: Appendix numbering is keyed to the associated chapters. Not all chapters have appendix 
material. 

Appendix 2A: Constructing the Sampling Frame 

Appendix 2B: Constructing the Matched Administrative Data Sample 

Appendix 2C: Constructing the Primary Data Collection Sample 

Appendix 4A: Analysis of Factors Related to Take-Up of the CEP 

Appendix 5 A: Analysis of Participation and Revenue 

Appendix 6A: Analysis of Impacts on Administrative Costs and Staffing 

Appendix 7A: Analysis of Impacts on Certification Errors 


pg. viii | Contents 


Abt Associates 



































Community Eligibility Provision Evaluation: Final Report 


Appendix 7B: Analysis of Impacts on Cashier Errors 
Appendix 7C: Analysis of Impacts on Meal Claiming Errors 
Appendix 8A: NSLP and SBP Meal Standards 

Appendix 8B: Menu Survey Processing for Meal Quality and Menu Variety Assessment 
Appendix 8C: Analysis of Impacts on Meeting Nutritional Standards and Meal Variety 


Abt Associates 


Contents | pg. ix 





Community Eligibility Provision Evaluation: Final Report 


Executive Summary 


Section 104(a) of the Healthy, Hunger Free Kids Act (HHFKA) of 2010 made the Community 
Eligibility Provision (CEP) available to Local Educational Agencies (LEAs) and schools in high 
poverty areas. Under the CEP, families are not required to submit applications for free or reduced- 
price (FRP) meals, and schools must provide free lunch and breakfast to all students. Schools are 
reimbursed for meals using a formula based on the percentage of students identified as eligible 
without an application (the identified student percentage, or ISP), using direct certification and other 
lists of eligible students. The potential benefits are to increase low income students’ access to 
nutritious meals, and to reduce the administrative burden LEAs experience with the school meals 
programs. 

The CEP is being phased in by the Food and Nutrition Service (FNS) of the U.S. Department of 
Agriculture (USDA) over a four-year period, beginning with three States selected to participate in the 
CEP during School Year (SY) 2011-12 (referred to hereafter as Year 1 States): Illinois, Kentucky, 
and Michigan. FNS chose four additional States for SY 2012-13 (Year 2 States): the District of 
Columbia, New York, Ohio, and West Virginia. FNS has selected four more States for SY 2013-14: 
Florida, Georgia, Maryland, and Massachusetts. The CEP will be available to all LEAs nationwide in 
SY 2014-15. 

As mandated by the HHFKA, the CEP Evaluation examined the implementation and impacts of this 
new framework for providing free meals to all students in high-poverty schools. The overall purpose 
of the evaluation was to obtain a better understanding of: 

• The acceptability of the Community Eligibility Provision to LEAs, 

• The incentives and barriers for LEAs adopting the provision, 

• Operational issues that State agencies encounter in administrating this provision, and 

• Implications and impacts of opting for this provision, including impacts on: 

- NSLP and SBP participation and revenues 

- LEA and school administrative costs and staffing 

- Errors in certification for FRP meals, counting reimbursable meals, and claims for 
reimbursement 

- Meal quality and choices. 

This Final Report presents the methods and results of the CEP Evaluation. It includes descriptive 
information on the implementation of the CEP from the State and LEA perspectives, and analytic 
findings on the factors affecting LEA participation in the CEP and its impacts on LEAs. 
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Study Design and Data 

The study design included an Implementation Study and an Impact Study in the first seven CEP 
States. For the Implementation Study, the evaluation team collected State administrative data on the 
characteristics of all LEAs and schools in three categories: participating in the CEP, eligible but non¬ 
participating, and near-eligible. (To be eligible for the CEP, an LEA must have at least one school 
with an ISP of 40 percent or more. LEAs with an average ISP of 40 percent or more across all schools 
are eligible to participate in the CEP on an LEA-wide basis.) The study fielded a survey of 1,322 
LEAs, including 360 participating, 671 eligible non-participating, and 291 near-eligible LEAs in the 
seven CEP States. In addition, State Child Nutrition Agency directors and staff were interviewed in 
2012 and 2013 to obtain their perspectives on implementation successes and challenges. Finally, the 
evaluation team interviewed Title I directors in all 51 State Educational Agencies to gather data on 
programs using FRP meals data and how these programs might be affected by the elimination of FRP 
meals applications under the CEP. 

The Impact Study collected and analyzed data on outcomes for matched samples of treatment 
(participating) and comparison (non-participating) LEAs. The two groups were matched on 
characteristics related to the likelihood that the LEA would take up the CEP, using propensity score 
matching techniques. All analyses used regression models to control for observed differences between 
treatment and control groups. To estimate impacts on NSLP and SBP participation and Federal 
reimbursements, the evaluation team collected and analyzed State administrative data from the fall 
months for SY 2009-10 through 2012-13 on 285 participating LEAs and 528 matched non¬ 
participating LEAs. For other impacts, the evaluation team conducted several data collections in a 
series of nested samples, including: 

1. a web survey of 141 treatment and 105 comparison LEAs that collected non-Federal revenue data 
for SY 2009-10 through 2012-13 and qualitative data about changes in participation and 
operations; 

2. two data collections in a sub-sample of 52 treatment and 48 comparison LEAs, 

a. interviews with LEA and school personnel on staffing and administrative costs, and 

b. reviews of a sample of direct certification and FRP meals application records for the analysis 
of impacts on certification errors; and 

3. in a sub-sample of 27 treatment and 25 comparison LEAs (selected from the sub-sample 
described above), three more intensive data collections, 

a. observations of cashiers to collect data on errors identifying reimbursable meals, 

b. review of school and LEA records to obtain data on meal claiming errors, and 

c. self-reported data from foodservice managers in the sampled schools about foods and 
beverages offered for the NSLP and the SBP during a selected week. 

All surveys and on-site data collection visits were conducted in the Winter and Spring of 2013. 

Implementation Study Results 

Three States—Illinois, Kentucky, and Michigan—and their LEAs began implementing the CEP in SY 
2011-12, and continued operating the CEP in SY 2012-13. Four additional States, including the 
District of Columbia, New York, Ohio, and West Virginia, and their LEAs implemented the CEP in 
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SY 2012-13. Four more States—Florida, Georgia, Maryland, and Massachusetts—were well- 
positioned to implement during SY 2013-14 despite numerous challenges. The first seven 
implementing States identified, notified and recruited LEAs to participate. In turn, the LEAs learned 
about the CEP, made decisions about whether to implement and in which schools, and communicated 
with schools and the community. Some States and LEAs had already noticed increased meal 
participation, and according to States, LEAs were excited about the CEP. The first seven States 
reported that key challenges at the State level were (1) the limited time during initial implementation 
to understand the CEP, make decisions about participation, and implement it, and (2) understanding 
and addressing the implications of the CEP for educational programs that use individual student FRP 
meals certification data. At the LEA level, the most frequently reported barriers included financial 
concerns and uncertainty about the impacts of the CEP on NSLP and SBP participation. Equity issues 
and operational challenges (especially for LEAs with both eligible and non-eligible schools) were 
also important concerns. Addressing the timing and information issues for States is a relatively 
straightforward problem. Flowever, telephone interviews with all 51 State Title I directors indicated 
that States routinely use FRP data for multiple education-related purposes, so the lack of such data 
under the CEP represents a widespread challenge. 

Across the first seven States, a total of 420 LEAs and 2,312 schools participated in the CEP in SY 
2012-13. On average, 32 percent of eligible regular and charter LEAs, and 29 percent of eligible 
schools, participated in the CEP. Participation rates for eligible regular and charter LEAs ranged 
considerably—from 24 percent in Michigan to 65 percent in West Virginia. Based on the number of 
near-eligible LEAs, it appears that the potential to increase the number of eligible LEAs was small in 
most States; only Illinois and Ohio had more than 50 near-eligible LEAs. 

Among participating LEAs eligible to participate LEA-wide, 92 percent offered the CEP at all 
schools, and 97 percent of schools offered the CEP, on average. On the other hand, among 
participating LEAs that were not eligible LEA-wide, 53 percent offered the CEP at all eligible 
schools, and on average, 79 percent of eligible schools offered the CEP. (A school is eligible for the 
CEP if the LEA is eligible to participate LEA-wide, if the school has an ISP of 40 percent or more, or 
if the school is grouped with other schools so that the ISP for the group is 40 percent or more.) 

Compared with eligible non-participating regular and charter LEAs, participating LEAs of these types 
had more students, on average, higher ISPs and FRP meals eligibility percentages, higher percentages 
of students in grades K-5, and higher percentages of students who are Black. Despite their larger 
average size, participating regular and charter LEAs were more often very small (with enrollments 
under 500 students); they also were more often urban and more often charter schools. These patterns 
were replicated in most (although not all) of the first seven States. 

Four factors were significantly associated with CEP participation by LEAs: ISP, enrollment, State, 
and charter status. The odds of participation rose dramatically with each additional 10 percentage 
points in the ISP, especially between 40 and 70 percent. The odds of participation also increased for 
LEAs with larger numbers of schools. Flowever, LEAs with below-average-size schools were more 
likely to participate, possibly because of differences in school-level eligibility. Although charter 
LEAs were more likely to participate than regular public LEAs, this finding was significant only in 
Ohio. The above relationships were observed after controlling for the other factors in the analytic 
model, so they may differ from the descriptive results (such as the participation rates by State 
computed without controlling for differences in LEA demographics). Evidence from State CN 
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interviews suggests that the differences by State and charter status may reflect how the CEP was 
implemented; differences in school-level eligibility may also have contributed to variation in 
participation across LEA types. 

Impact Study Results 

As discussed in Chapter 2, the Impact Study used a matched comparison design, and impact estimates 
were computed from differences in outcomes between the treatment group (participating LEAs and 
schools) and the comparison group (non-participating LEAs selected to be similar in likelihood of 
CEP take-up, and selected schools in these LEAs). Outcome data were drawn from State 
administrative records, a web survey, in-person interviews and other on-site data collection, with the 
largest samples devoted to the participation and revenue impacts. 

While the impact estimates control statistically for known differences between the groups, there are 
some caveats to the findings, and the results must be interpreted with appropriate caution. These 
estimates, at best, measure the effect of the CEP on the specific group of LEAs that were observed: 
those that chose to participate in the first two years of availability in the Year 1 States (Illinois, 
Kentucky and Michigan) and those that chose to participate in the first year of availability in the three 
Year 2 States (New York, Ohio, and West Virginia). (The District of Columbia was excluded because 
the matched comparison design could not be used there.) These estimates do not generalize to LEAs 
in these States that are not comparable to the participating LEAs, nor to LEAs in other States. The 
matching and statistical controls do not account for potentially important, unmeasurable 
characteristics that determine the take-up decisions of LEAs, such as the perception that the LEA will 
benefit from adopting the CEP. Nevertheless, the study used the most rigorous and feasible methods 
to separate the impact of the CEP from the other factors that could have affected the outcomes of 
interest. The results for impacts on participation and revenues for NSLP and SBP are based on the 
strongest component of the evaluation, which uses a time-series of pre- and post-implementation data 
to strengthen the controls for differences in relevant characteristics between the treatment and control 
groups. Other impact results are based on a posttest-only comparison design and should be considered 
exploratory. 

Judged by the confirmatory outcome selected for the evaluation, the CEP was clearly successful: it 
significantly increased student participation in NSLP and SBP, and Federal reimbursements per 
reimbursable meal for these programs were significantly greater among the LEAs that chose to 
participate. The CEP’s impact on student participation in school meal programs was surprisingly 
consistent across programs, increasing participation by 5 to 9 percent (relative to the lunch and 
breakfast programs’ participation in the matched comparison schools, after controlling for differences 
between the two groups). The CEP’s impact on Federal reimbursements per meal was a significant 
increase of about 6 percent for the NSLP and 2 percent for the SBP. Total Federal revenue per student 
increased by 13.5 percent. On the other hand, the CEP did not appear to have an impact on any of 
type of non-Federal revenue. Taken together, the finding of increased Federal funding and the lack of 
evidence that other revenues significantly declined suggest that the CEP did not have an adverse 
overall effect on LEA foodservice revenues, and may have produced a net gain for participating 
LEAs. 

The CEP requires LEAs to offer the SBP, and it was available in all schools in nearly all participating 
LEAs and non-participating LEAs that are likely to take up the CEP. Study data did not indicate that 


pg. 4 1 Executive Summary 


Abt Associates 





Community Eligibility Provision Evaluation: Final Report 


the CEP had an impact on availability of the SBP. After controlling for differences between groups, 
participating LEAs used traditional line service less often at breakfast and served breakfast in the 
classroom more often, as is recommended to increase SBP participation (USDA, 2013e) but 
complicated to implement under conventional meal counting and claiming procedures. Participating 
LEAs were less likely to offer a choice of foods at breakfast. The reduction in choice of foods may be 
linked to the greater use of in-classroom breakfast. 

As expected, CEP participation appeared to reduce time spent by school and LEA staff on distributing 
and processing applications for free or reduced-price meals, verifying income of free/reduced price 
students, and meal payment collections and accounting. The combined savings for these activities 
were 68 minutes per student per year, representing labor cost savings of about $29 per student per 
year. The CEP had no impact on the staff time and costs associated with direct certification. On the 
other hand, the CEP appeared to increase the staff time and cost of counting and claiming 
reimbursable meals from less than 30 minutes per student per year to almost an hour, offsetting half 
of the savings in other activities. Possible reasons for this result include: increased participation in 
school meal programs, and new (and sometimes more manual) meal counting and claiming 
procedures. It is too early to tell whether the increased time for meal counting and claiming reflects 
an early implementation challenge, and adaptation to the new procedures and planned automation by 
States might reduce this impact over time. On the other hand, the CEP could produce net savings in 
meal counting and claiming costs in the long run, if schools could reduce the complexity and cost of 
automated systems for meal counting and payments. Treatment and comparison schools did not differ 
in the total time and cost spent on administrative tasks overall or for any category of worker. 
Descriptive data from interviews and surveys suggest that, for the majority of LEAs, the CEP did not 
require LEAs to increase staff, the time spent on meal counting and claims was not a concern, and the 
CEP made it easier for LEAs to break even or at least had no impact on the foodservice bottom line. 

The CEP reduced the overall rate of certification errors and had little or no impact on errors in 
counting meals (at the cashier level) and claiming meals for reimbursement. The CEP did not appear 
to affect errors in direct certification, which are rare. The CEP eliminated application processing 
errors that, in the comparison schools, resulted in approximately 6.6 percent of applicants having the 
wrong certification level (too high or too low). The CEP also eliminated procedural errors in 
application processing that, in the comparison schools, affected 20.2 percent of approved applicants 
and 16.9 percent of denied applications. Procedural errors (such as missing certification documents 
or incomplete applications) do not necessarily result in certification errors but indicate a lack of 
integrity in the process. The CEP had no significant impact on cashier error in identifying 
reimbursable meals, despite concerns that increased participation might lead to more error by 
speeding up service lines. One measure of meal claiming error was significantly higher in 
participating schools, but there was no impact on overall meal claiming error. The primary reason for 
errors in claims submitted for CEP schools to the State was that the LEA used the wrong claiming 
percentage, and usually the claiming percentage used was lower than the LEA was entitled to use. 

This problem might be readily addressed through training or through claims processing by the State 
(which may have corrected some or all of the detected errors in claiming percentages). The baseline 
levels of claiming error in the NSLP and SBP appear to be quite small, at least in the sample, so the 
lack of a positive impact from the CEP is not a reason for concern. 

For a broad range of meal quality outcomes, there was no evidence that the CEP had a significant 
impact. The methods used by the study to assess meal quality emulated, to extent practical, the 


Abt Associates 


Executive Summary | pg. 5 




Community Eligibility Provision Evaluation: Final Report 


methods used by FNS and State Agencies to measure compliance with the meal component standards. 
However, the study simplified the approach to assessing schools’ compliance with meal component 
standards in order to reduce burden on participating schools. Therefore the research team did not 
have access to all of the information States routinely use for this puipose. While the study’s 
simplified methods allowed objective comparisons of CEP and non-CEP schools on meal 
components, the study results should not be interpreted as representing the same results FNS or State 
agencies would generate when they assess compliance with NSLP standards. The study’s approach 
likely resulted in a more conservative assessment of school meals. 

At lunch, CEP schools tended to offer more vegetables but were less likely than comparison schools 
to have met the (soon-to-be-required) Target 1 sodium standard. There was no evidence of impacts on 
meeting NSLP requirements for other food components, the calorie range requirement, or the percent 
of calories from saturated fat. There were no significant differences in meeting food component 
requirements or the Target 1 sodium standard for breakfasts offered. 

The CEP appeared to have little if any impact on the number of choices offered in meal components. 
There were no significant differences in the number of choices offered for fruits, vegetables, grain, 
milk and entrees for lunches between the treatment and comparison schools, nor was there a 
significant difference in choices for breakfast components. For lunch, the only significant impact of 
the CEP was a small increase in the number of desserts offered, although the number of desserts 
offered was minimal and well within allowed limits for both CEP and comparison schools. Changes 
in the types of food offered to serve more quickly, and shifts to breakfast in the classroom in CEP 
schools were reported but had no significant effect on the number of breakfast choices offered in CEP 
schools. 


Summary 

Summing up across all of the evaluation results, the implementation of the CEP in its first two years 
was successful in perhaps the two most important dimensions. First, the take-up of the CEP was 
widespread among eligible LEAs, despite the uncertainties about impacts on finances and operations; 
further, participating LEAs were both well-satisfied and likely to continue using the CEP. Second, the 
CEP appeared to increase NSLP and SBP participation and the associated Federal reimbursements, 
while not adversely affecting the financial bottom-line of foodservice operations. The CEP eliminated 
substantial levels of error in the processing of FRP meals applications, as well as eliminating the costs 
of processing and verifying these applications. For most other outcomes, there were little or no 
impacts, either positive or negative. 

The results point to several key challenges for the future of the CEP. First, there is substantial 
skepticism about the feasibility of the CEP among eligible non-participating LEAs; wider adoption of 
the CEP may require some efforts to address this skepticism. Second, the need for household income 
data for Federal and State educational programs will continue to pose perhaps the greatest 
implementation challenge. Third, impact findings suggest that the LEAs experience new challenges in 
meal claiming under the CEP, at least during initial implementation, based on the evidence of 
increased administrative costs and errors. Several currently-implementing CEP States have already 
installed automated systems to prevent errors in the use of claiming percentages, and others indicate 
that they plan to do so. As this practice becomes more widespread, it would presumably eliminate the 
main source of claiming error observed in CEP schools. Finally, the findings on meal quality and 
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choices provide evidence of little change in meal quality, suggesting that the participating LEAs have 
not reduced meal quality to contain costs but also that any resources freed up by the CEP have not led 
to broad improvements in meals. The only significant effects on meal quality were the positive 
impact on vegetables offered and the negative impact on meeting the future intermediate sodium 
target. Given the timing of the study, it is too early to determine whether the observed effects on meal 
quality are merely transient (reflecting CEP implementation, introduction of new standards, or both) 
or likely to be sustained. Across the multiple outcomes assessed by the Impact Study, the clear and 
positive impacts on NSLP and SBP participation and reimbursements represent the most notable 
results of implementing the CEP. 
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1. Introduction 


Section 104(a) of the Healthy, Hunger Free Kids Act (HHFKA) of 2010 made the Community 
Eligibility Provision (CEP) available to Local Educational Agencies (LEAs) and schools in high 
poverty areas. 1 Under the CEP, families are not required to submit applications for free or reduced- 
price meals, and schools must provide free lunch and breakfast to all students. Schools are reimbursed 
for meals using a formula based on the percentage of students identified as eligible without an 
application (the identified student percentage, or ISP), using direct certification and other lists of 
eligible students. 2 The potential benefits are to increase low income students’ access to nutritious 
meals, and to reduce the administrative burden LEAs experience with the school meals programs. 

The CEP is being phased in over three years. The Food and Nutrition Service (FNS) of the U.S. 
Department of Agriculture (USD A) selected three States to participate in the CEP during School Year 
(SY) 2011-12 (referred to hereafter as Year 1 States) and four additional States for SY 2012-13 
(Year 2 States). FNS has selected four more States for SY 2013-14 (Year 3 States), and the CEP will 
be available to all LEAs nationwide in SY 2014-15. 

As mandated by the HHFKA, the CEP Evaluation examined the implementation and impacts of the 
new framework for providing free meals to all students in high-poverty schools. The overall purpose 
of the evaluation was to obtain a better understanding of: 

• The acceptability of the Community Eligibility Provision to LEAs, 

• The incentives and barriers for LEAs adopting the provision, 

• Operational issues that State agencies encounter in administrating this provision, and 

• Implications and impacts of opting for this provision. 

This Final Report presents the methods and results of the CEP Evaluation. It includes descriptive 
information on the implementation of the CEP from the State and LEA perspectives, and analytic 
findings on the factors affecting LEA participation in the CEP and its impacts on LEAs. 

The next section discusses the background of the CEP in the overall context of the structure of school 
meal administration, the mandate for an evaluation, and an overview of the evaluation design. The 
rest of the chapter discusses the research objectives of the evaluation and concludes with the scope 
and organization of this report. 


1 During the evaluation’s planning and data collection phases, the CEP was known as the Community 
Eligibility Option (CEO). 

2 Eligibility can also be certified based on lists maintained by LEAs or other agencies that identify 
categorically eligible students, such as homeless, runaway and migrant youth, and foster children. 


Abt Associates 


1. Introduction | pg. 9 







Community Eligibility Provision Evaluation: Final Report 


1.1 Study Background 

Two school-based programs—the National School Lunch Program (NSLP) and School Breakfast 
Program (SBP)—are vital parts of the safety net against childhood hunger overseen by FNS. Each 
State administers these programs through Local Educational Agencies (LEAs) 3 and is responsible for 
ensuring that the LEAs meet Federal program guidelines. Students in households with family incomes 
at or below 130 percent of the Federal poverty line (FPL) are eligible for free meals at school, while 
those in households with family incomes between 130 percent and 185 percent of FPL are eligible for 
reduced price meals. The NSLP and SBP provide foods that meet the government’s nutritional 
standards for two meals each day that school is in session. 

More than 90 percent of students nationwide attend schools that offer NLSP and/or SBP. In Fiscal 
Year 2012, over 31 million children participated in the NSLP, and almost 13 million children 
participated in the SBP. Over 68 percent of lunches and 84 percent of breakfasts served in schools 
were for children eligible for free or reduced price (FRP) meals (USDA, 2013a, 2013b). Eligibility 
for FRP meals is determined through annual household applications, direct certification, and lists of 
eligible children. Households can apply based on income or categorical eligibility (i.e., being part of a 
group that is specified as eligible by law regardless of income). 

Direct certification determines children’s eligibility for free meals by matching student enrollment 
records with administrative records from programs that make children categorically eligible for free 
meals, including the Supplemental Nutrition Assistance Program (SNAP), Temporary Assistance for 
Needy Families (TANF), or the Food Distribution Program on Indian Reservations (FDPIR). In 
addition, children can be certified as eligible for free meals based on lists of other categorically 
eligible individuals maintained by LEAs or other agencies, including homeless, runaway and migrant 
youth, and foster children. Children certified for free meals by any of these means—without 
applications—are referred to as “identified students.” 

The Child Nutrition and WIC Reauthorization Act of 2004 required States to develop a direct 
certification system for school-age SNAP participants by SY 2008-09. During SY 2011-12, 89 
percent of LEAs participating in the NSLP directly certified some SNAP participants. Nationally, 


The Richard B. Russell National School Lunch Act (NSLA) uses two different terms to refer to the local 
entities that enter into agreements with State agencies to operate the school meal programs. The Child 
Nutrition and WIC Reauthorization Act of 2004 (P.L. 108-265) amended the NSLA by using the term 
Local Educational Agency (LEA), defined for public schools in the Elementary and Secondary Education 
Act of 1965 (ESEA), when referring to the application, certification, and verification functions of the 
school meal programs. Sections of the NSLA that deal with other aspects of the programs, such as meal 
pattern requirements and meal-counting and claiming reimbursements, use the term School Food Authority 
(SFA), which current NSLP regulations define as the governing body that has the legal authority to operate 
the NSLP/SBP in one or more schools. The term most commonly used for the entities described as LEAs in 
the ESEA is school districts. While this definition applies only to public entities, State agencies also enter 
into agreements to operate the NSLP with charter schools, non-public schools, or other non-profit local 
entities such as an archdiocese running multiple non-public schools. Because the section of the HHFKA 
that authorizes the CEP refers to LEAs, that term is used throughout this document to refer to both public 
and non-public nonprofit local entities that enter into agreements with State agencies to operate the NSLP 
and SBP. 
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direct certification identified 86 percent of school-age SNAP participants. This measure of 
effectiveness in reducing the need for household applications increased 17 percent over the previous 
year (2010-11) (Moore, Conway, and Kyler, 2012). 

LEAs participating in NSLP and SBP receive Federal reimbursements for meals to eligible children. 
Under conventional procedures, LEAs separately count reimbursable meals served to (a) children 
certified for free and reduced price meals and (b) children paying the LEA’s full price (“paid” meals). 
LEAs submit monthly claims for reimbursement based on meal counts and rates for free, reduced, and 
paid meals set by USDA, to their respective State Child Nutrition (CN) Agencies. 

To reduce local-level administrative burdens that accompany completing and processing the 
household applications, USDA has established three provisions as alternatives to the traditional 
requirements for annual determinations of FRP meal eligibility. All three provisions let LEAs conduct 
FRP meals certification less frequently than annually: every two years under Provision 1, and at most 
every four years under Provisions 2 and 3. These provisions can be used throughout the LEA or 
specifically for individual schools. Under Provisions 2 and 3, LEAs offer free meals to all students 
and count only the total meals, then claim reimbursement using percentages or dollar amounts 
established in the base year when students are certified for FRP meals (USDA, 2013c). 4 LEAs must 
still collect applications every four years unless they receive an extension for an additional four years. 
Stopping and starting the application process can represent a challenge for families, schools, and 
LEAs, who may become out of practice with the application process, especially if four-year 
extensions have been granted (National Research Council, 2012). 

1.1.1 Authorization for the CEP and Mandate for the Evaluation 

Recognizing the problem of both childhood obesity and food insecurity in children, Congress passed 
the E1HFKA in 2010. The HE1FKA authorized funding and reformed policies for the USDA child 
nutrition programs, including NSLP and SBP, with the goal of improving the quality of and 
increasing access to meals offered to children. The E1E1FKA updated standards for nutritious meals 
and provides additional funding for healthier meals. Several other program changes were designed to 
improve access to NSLP and SBP through program expansion and outreach, including the CEP. As 
part of the E1E1FKA, Congress further mandated that FNS conduct an evaluation (1) to understand 
how the CEP is implemented; (2) to describe incentives and barriers for LEA and school 
participation; and (3) to estimate impacts on LEAs, schools and children. 

1.1.2 Purpose and Design of the CEP 

Section 104(a) of the HHFKA created the CEP for LEAs and schools as an alternative to household 
applications for FRP meals and conventional reimbursement methods. The CEP can be used in an 
entire LEA (district-wide), a group of schools in an LEA, or an individual school —if the ratio of 
identified students (i.e., those certified for free meals without application, as explained in Section 1.1) 
to total enrollment is 40 percent or more in the year prior to adopting the CEP (known as the year 
prior to year 1 of CEP operations). 5 This criterion is known as the “Identified Student Percentage” 


4 http://www.fns.usda.gov/cnd/governance/prov-l-2-3/Provl_2_3_FactSheet.htm 

5 Schools participating in Provision 2 or 3 may elect the CEP if (a) they are not in the base year, when direct 
certification must be conducted, and (b) they can provide an ISP for the year prior to the first year of the 
CEP. 
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(ISP). LEAs and schools using the CEP must agree to serve both breakfasts and lunches free to all 
students, and to cover all costs of these meals in excess of Federal reimbursements with non-Federal 
funds. The percentage of meals reimbursed with Federal funds at the free meals rate (the “claiming 
percentage”) equals the ISP multiplied by a factor of 1.6, as established in the HHFKA; all other 
meals are served free to students but are reimbursed at the paid meals rate. The applicable ISP is for 
the entire LEA, the group of schools, or the individual school, depending on how the LEA chooses to 
implement the CEP and establish CEP eligibility for its schools. 

The CEP has two potential advantages, compared to conventional reimbursement methods: (1) it can 
potentially increase student participation in meal programs by expanding access to free meals for all 
students, and (2) it can decrease household and administrative burden by not requiring regular 
applications to establish eligibility, and by simplifying the counting of reimbursable meals. 

The CEP allows four years of reimbursement to eligible LEAs and schools with an additional four- 
year cycle possible if the LEA or school continues to meet the minimum ISP of 40 percent. LEAs 
using the CEP are encouraged, but not required, to conduct direct certification. If the LEA chooses to 
conduct direct certification (or if the State conducts direct certification) and the ISP increases over the 
course of participation, the LEA will be reimbursed at a higher claiming percentage. 6 An LEA or 
school may withdraw from the CEP at any time during the four-year cycle. 

The phase-in of the CEP has followed a schedule specified by the HHFKA. FNS initially identified 
10 States with the greatest potential for covering the highest number of qualifying LEAs and schools 
to apply for the CEP for the first year, SY 2011-12. FNS selected three States (Illinois, Kentucky, and 
Michigan), all of which offered the Provision in SY 2011-12. 7 For SY 2012-13, FNS encouraged all 
States to apply for the CEP, and selected three additional States (New York, Ohio, and West Virginia) 
and the District of Columbia, making a total of seven participating States. 8 FNS has similarly selected 
another four States to participate in the CEP for SY 2013-14 (Florida, Georgia, Maryland, and 
Massachusetts). The CEP will be available nationwide to all eligible LEAs and schools in SY 2014- 
15. 


1.2 Study Objectives and Overview 

The evaluation comprised two study components: the Implementation Study and the Impact Study. 
The Implementation Study provides a description of the scope and operations of the CEP, and offers 
lessons from the first seven States to implement the CEP. The Impact Study provides estimates of the 
impacts of the CEP on school meals program participation and revenues, foodservice staffing and 
costs, program integrity, and the nutritional quality of meals offered. This section provides an 
overview of the study objectives and research questions. 


6 To renew participation after the fourth year, LEAs must obtain identified student data demonstrating that 
they still meet the ISP requirement for the participating schools. 

7 Although FNS originally selected Tennessee among the three States, it subsequently opted out, and was 
replaced by Michigan. 

s For ease of discussion, the term “State” in this report includes the District of Columbia (D.C.) government, 
which supervises all LEAs participating in the NSLP and SBP within its borders, including the D.C. Public 
School system, public charter schools, and non-public schools. 
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1.2.1 Implementation Study Objectives 

The Implementation Study had three research objectives to fulfdl the Congressional mandate. 

1: Estimate the number of eligible LEAs and schools that have not chosen to participate in the CEP, 

2: Assess the barriers to participation in the CEP in non-participating but eligible LEAs and schools, 
and 

3: Describe the LEAs and schools participating in the CEP. 

These objectives required two distinct types of data. To meet Objectives 1 and 3, the study used 
descriptive data on three groups of LEAs and schools in School Year (SY) 2012-13: 

• Those that were participating in the CEP, 

• Those that were eligible but non-participating, and 

• Those non-participating LEAs that were near-eligible (i.e., those with ISPs close to but not above 
the minimum eligibility level of 40 percent). 

To meet Objective 2, the study collected data on the experiences and attitudes of the 11 CEP States 
and of the LEAs in the Year 1 and Year 2 States that constituted the three groups of interest. These 
data included not only perceptions of barriers but also facilitators of the CEP. In addition, the study 
used the descriptive data in statistical modeling to determine the LEA characteristics related to CEP 
participation. Finally, the study collected data from 51 State Education Agencies (SEAs) on one of 
the key barriers to participation: the impact of the CEP on the processes of allocating Federal and 
State educational funds at the LEA and school levels. 

1.2.2 Impact Study Objectives and Research Questions 

The Impact Study was designed to meet Research Objective 4 by estimating the impacts of the CEP 
on five domains of school meals program outcomes: participation and revenues, availability of the 
SBP, LEA administrative costs, program integrity, and meal quality. The research questions for these 
domains are discussed below. 

Program Participation and Revenue Impacts 

The two outcomes most central to evaluating the CEP’s success are impacts on participation and 
revenues. The specific research questions regarding these outcomes are: 

• How did average daily participation in the NSLP change because of the CEP (compared to what 
participation would have been in the absence of the CEP)? 

• How did Federal revenues per reimbursable meal change because of the CEP (compared to 
revenues that would be generated from the traditional meal reimbursement structure)? 

Additional questions for this impact domain include the following: 

• How did average daily participation in the SBP change because of the CEP (compared to what 
participation would have been in the absence of the CEP)? 

• To what extent (if any) did the impacts of the CEP on program participation: 
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- Reflect changes in the number of actual participants, frequency of participation, or both, for 
NSLP, SBP, and total meals? 

- Vary by grade level for NSLP, SBP, and total meals? 

• How did revenues from non-Federal sources, including State funds, local funds, student payments 
for reimbursable meals, and payments for competitive foods 9 and adult foods, change because of 
the CEP compared to revenues that would be generated from the traditional meal reimbursement 
structure? 

• How do revenues under the CEP compare to revenues that would be generated under other 
special assistance alternatives (Provision 1, 2, or 3)? 

• How much local funding is needed compared to these other alternatives? 

The research questions for the other four domains are listed below. 

School Breakfast Program Impacts 

• How did the CEP impact the School Breakfast Program in terms of whether schools offer 
breakfast and the type of breakfast service offered? 

• To what extent did the CEP affect the foods served at breakfast, specifically: 

- Whether the identical breakfast is served to all students (or more generally, whether the CEP 
affects the amount of breakfast food choices for students)? 

- Whether and to what extent a la carte and competitive foods are offered during breakfast? 
Staffing and Cost Impacts: 

• To what extent did the administrative burden of operating the NSLP and SBP change under the 
CEP? 

• What is the difference in administrative costs to the LEA associated with the CEP compared to 
the costs in the absence of the CEP? 

• To what extent did the costs of producing reimbursable meals change under the CEP? 

Program Integrity Impacts: 

• How did the CEP impact program integrity, specifically: 

- What is the impact on errors associated with the certification process, including 
overcertification, undercertification, and procedural errors? 

- What is the impact on errors in meal counting by cashiers and in computing claims for 
reimbursement? 


Q Here the term “competitive foods” includes all foods sold to students by school foodservice other than 
reimbursable meals. As such, this does not include food sales by other entities, including the athletic 
department or school store. 
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Meal Quality Impacts: 

• To what extent, if any, did the nutritional profile of school meals offered change under the CEP 
(compared to what would have happened in the absence of the CEP)? 

• To what extent, if any, did the variety of food choices offered to students in school meals change? 

• To what extent, if any, did compliance with USDA-FNS nutrition standards change? 


1.3 Overview of Study Samples 

This section describes the universe of LEAs and the samples for the Implementation Study and the 
Impact Study, and the relationship among the samples selected for each study and data collection 
component. The elements of the Implementation and Impact Studies were nested, such that as data 
collection demands became more intensive, successively smaller samples were required, thereby 
reducing the burden on participating LEAs, while providing important contextual information about 
those LEAs selected for the most intensive on-site data collection activities. Exhibit 1.1 illustrates the 
nested relationships of the universe and samples. The data sources are described in Chapter 2. 

1.3.1 Administrative Data for the Universe of LEAs 

To identify LEAs eligible or near-eligible for the CEP, and to describe their characteristics, the 
Implementation Study collected administrative data on all LEAs and schools in the seven States 
operating the CEP in SY 2012-13. 10 Within this universe, additional administrative data were 
collected on participating LEAs and schools. 

The universe in the seven CEP States included 419 participating, 834 eligible non-participating, and 
438 near-eligible LEAs. This was the universe for the Implementation Study LEA survey sample. 

1.3.2 Implementation Study LEA Survey Sample 

To meet the broad range of Implementation Study objectives, the study fielded a survey of 1,322 
LEAs, including 360 participating, 671 eligible non-participating, and 291 near-eligible LEAs in the 
seven CEP States. Chapter 2 of this report describes the process of identifying LEAs and selecting 
respondents in each of these groups. 


10 In early 2014, the evaluation will report on the numbers and characteristics of participating, eligible non¬ 
participating, and near-eligible LEAs in the 11 States operating the CEP in SY 2013-14, using 
administrative data from these States. 
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Exhibit 1.1: Graphical Depiction of Relationship across Implementation and Impact Studies 
and Sample Sizes 



Legend: NE: near-eligible LEAs; EN: eligible non-participating LEAs; P: participating LEAs. 

T: treatment (participating) LEAs for the Impact Study; C: comparison (non-participating) LEAs for the Impact 
Study. 

a Administrative data include all LEAs in the State records on LEA participation counts and revenues for the 
NSLP and the SBP for the following States: Illinois, Kentucky, Michigan, New York, Ohio, West Virginia. 
Administrative data from all seven CEP States including the District of Columbia were used to identify the 
universe in the largest circle. 

b The universe of near-eligible, eligible non-participating and participating LEAs is defined as all LEAs that 
appear in the 2011-12 FNS Verification Summary Report (VSR) data set and either (a) participate in the CEP, 
(b) are eligible for the CEP, or (c) are nearly eligible for the CEP. See Section 2.1 for description of the VSR 
data. 
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1.3.3 Matched Administrative Data Sample for the Participation and Revenue Analysis 

Administrative data on NSLP and SBP participation and Federal revenues were collected for all 
LEAs in the six States in the Impact Study: Illinois, Kentucky, Michigan, Ohio, New York and West 
Virginia; note that the District of Columbia was included in the Implementation Study but not the 
Impact Study because of a lack of suitable comparison LEAs, as discussed in Chapter 2. From the 
universe of LEAs in these States, the evaluation team selected a sample of 525 non-participating 
LEAs to match with 286 participating LEAs for the primary analysis of the impacts of the CEP on 
NSLP and SBP participation, and for analysis of impacts on Federal revenues for school meals. This 
approach allowed the study to reduce the burden of the evaluation on LEAs selected for the Impact 
Study. 

1.3.4 Component 1: Participation, Enrollment, Attendance and Revenue (PEAR) Survey 

Component 1 was the largest of three primary data collection components for the Impact Study. In 
this component, 144 treatment (participating) LEAs and 108 matched comparison (non-participating) 
LEAs completed a web survey and provided LEA-level participation, enrollment, attendance, and 
revenue data to complement the State administrative data. The LEA sample for Component 1 was 
then sub-sampled for Component 2 (The matching approach for Components 1, 2, and 3 is discussed 
in Section 2.3.) 

1.3.5 Component 2: Administrative Costs and Certification Errors Data 

Component 2 comprised data collection on two domains: administrative costs and certification errors. 
These data were collected in a matched sample of 52 treatment LEAs and 51 comparison LEAs. Data 
for Components 2 and 3 were collected at both the LEA and school levels, with up to three sampled 
schools per LEA. All LEAs selected for Component 3 were included in the sample for Component 2. 

1.3.6 Component 3: Meal Quality and Meal Counting/Claiming Data 

Component 3 included data collection on two domains: meal quality and errors in counting and 
claiming reimbursable meals. These data were collected in a matched sample of 27 treatment and 25 
comparison LEAs, including 2 or 3 schools in each LEA. 

1.4 Scope of the Report 

Chapter 2 of this report describes the methods and data for the Implementation and Impact Studies, 
including study design, sampling, administrative sources, and primary data collection. Chapter 3 
addresses the Implementation Study objectives by providing a description of the implementation 
process, the challenges and lessons learned, and the incentives and barriers to participation, and 
information from a survey of State Education Agencies on issues related to the use of FRP meals 
eligibility data by educational programs. Chapter 4 compares the participating LEAs and schools with 
eligible non-participating and near-eligible LEAs and schools; it also presents a statistical analysis of 
the factors related to LEA participation in the CEP. Chapter 5 presents the analysis of the impact of 
the CEP on NSLP and SBP participation by students. Chapter 6 presents the analysis of the impact of 
the CEP on LEA Staffing and Administrative Costs. Chapter 7 provides the results of the analysis of 
Program Integrity Impacts (certification, cashier, and meal counting/claiming error). Chapter 8 
presents the impact of CEP on nutritional quality and choice and variety in meals offered in on NSLP 
and SBP. Chapter 9 provides a discussion of the overall findings and conclusions. 
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2. Study Design, Sampling, and Data 


This chapter discusses the study’s design, sampling, and data sources. Section 2.1 discusses the 
Implementation Study, and Sections 2.2 through 2.4 describe the Impact Study. Further details of 
design, data, and methods are presented in subsequent chapters, in the context of the discussion of the 
specific analyses and results. As explained in Chapter 1, the primary data collection for the 
Implementation Study was nested within the universe of LEAs in the seven CEP States, and the 
primary data collection for the Impact Study was nested within the LEA sample for the 
Implementation Study in six of the seven CEP States (excluding the District of Columbia). 
Administrative data from State records and extant national databases were collected for all LEAs for 
use in both parts of the evaluation. 

2.1 Implementation Study Design, Sampling, and Data 

The Implementation Study was designed to address three research objectives: (1) to estimate the 
number of eligible LEAs and schools that do not choose the CEP, (2) to assess the barriers to 
participation in the CEP in eligible but non-participating LEAs and schools, and (3) to describe the 
LEAs and schools participating in the CEP. The Implementation Study used three sets of data: State 
Child Nutrition Director Interviews, State and national databases, and web surveys of LEAs. These 
data sources and the data collection methods for the Implementation Study are described below. 

2.1.1 Design and Sampling 

The sampling and data collection for the Implementation and Impact Studies used a specific set of 
definitions of the three types of LEAs of interest based on LEA and school-level information, 
following the FNS guidance: 

1) participating in the CEP: LEAs with at least one participating school 

2) eligible non-participating: LEAs with at least one eligible but no participating schools 

3) near-eligible non-participating: LEAs with a district-wide ISP between 30 and 40 percent and no 
eligible schools. 11 

Exhibit 2.1 su mm arizes how the three groups of LEAs were identified. In the five States where data 
were available, 12 the evaluation team used the official definition of LEA eligibility (as discussed in 
Section 1.1.2); consequently, the “eligible” category includes all LEAs with an ISP over 40 percent, 
plus LEAs with an ISP below 40 percent with at least one eligible school. In these States, the “near- 


11 The near-eligible non-participating LEAs are defined on the basis of the requirement that States notify 
these LEAs about the CEP. 

12 All seven States provided data on participating LEAs and schools, including ISPs. Five States—all but 
Ohio and New York—provided statewide ISP data drawn from April 2012 at both the LEA- and the 
school-levels, which allowed the evaluation team to identify these five States’ participating and eligible 
non-participating LEAs. The evaluation team used the State-provided list of LEAs Ohio and New York had 
identified as eligible in their applications to FNS to participate in the CEP. It was these LEAs that the 
States had notified about the CEP availability. Under this definition, LEAs with an ISP over 40 percent that 
did not appear on the State eligible list were included in the near-eligible category. 
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eligible” category comprised the LEAs with ISPs in the 30 to 40 percent range with no eligible 
schools. Exhibit 2.1 explains the alternate definition of LEA eligibility used where school-ISPs for 
non-participating LEAs were not known. 

Exhibit 2.1: Definitions of LEA Eligibility and Participation in CEP 

Type of CEP 
Eligibility/ 
Participation 

FNS Definition 

Operationalized Definition 

Participating 

The LEA has at least one 
participating school 

The LEA has at least one participating school 
reported by the State Child Nutrition Agency 

Eligible non¬ 
participating 

The LEA has at least one eligible 
school (as specified by FNS, i.e., 
with ISP at least 40%) but no 
participating schools 

(a) The LEA has at least one school with an 

ISP of 40% or higher as of April 1,2012 
according to State-provided data (DC, IL, KY, 

Ml, WV); or 

(b) The LEA appeared on the list of eligible 

LEAs that the State submitted to FNS as part 
of its application to participate in the CEP 
(NY, OH). 

Near-eligible 

The LEA has an ISP of at least 30 
percent but less than 40 percent 
and no eligible schools 

The LEA has a projected ISP of 30 percent or 
higher (according to calculations using 

October 2011 VSR data) and is not eligible or 
participating. 3 


a Projected ISP = Number Free Eligible Not Subject to Verification/Enrollment. This measure is “projected” in the 
sense that it is based on October 2011 data rather than the April 2012 data that States were required to use 
under FNS guidance. April 2012 ISP data were not available for all LEAs in all States, so October 2011 data 
were used. 

The CEP participation and eligibility classification of schools for the evaluation generally follows the 
LEA classification scheme and the FNS rules, with some modifications to accommodate data 
limitations. All schools in near-eligible LEAs were classified as near-eligible, and all schools in 
eligible non-participating LEAs were classified as eligible non-participating (based on the assumption 
that schools with ISPs below 40 percent could be combined with schools with ISPs above 40 
percent). However, non-participating schools in participating LEAs were classified as follows: (1) 
eligible non-participating if they either belonged to an LEA with an ISP of 40 percent or more (and 
therefore all schools in the LEA were eligible regardless of ISP) or had a school ISP of 40 percent or 
more; (2) near-eligible otherwise (regardless of ISP, because of the possibility of combinations with 
schools with higher ISPs to make a near-eligible group). These definitions were used to ensure 
consistent classifications for all States. lj 


13 The school ISP data needed to construct school-specific CEP participation and eligibility classification 
were completely missing for two States (New York and Ohio) and partially missing for two others (Illinois 
and Michigan). If the ISP was missing for a non-participating school in a participating LEA, it was 
classified as eligible non-participating. 
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2.1.2 Administrative and Extant Data 

The evaluation team constructed the database for the descriptive analysis of LEA and school 
characteristics from several sources. The universe of LEAs for the analysis was constructed from the 
FNS Verification Summary Report (VSR) data. 14 The States provided their administrative data to 
identify participating LEAs and eligible non-participating LEAs from among those listed in the VSR 
data. States also provided data to identify participating and eligible non-participating schools. The 
evaluation team combined the VSR and State data with LEA and school characteristics from the 
Common Core of Data (CCD) to complete the analysis database. 15 These data sources and their 
associated issues are described in Appendix 2A. The analysis of these data is presented in Chapter 4. 

2.1.3 LEA Web Surveys for Implementation Study 

As part of the Implementation Study, the evaluation team conducted a web-based survey in January to 
June 2013 to collect information on LEA/school characteristics and environments as well as the 
incentives and barriers to adopting and implementing the CEP. The primary survey respondent was 
intended to be the LEA foodservice director (FSD), although other central foodservice personnel were 
sometimes called upon to respond to some survey questions. To obtain the target sample for the 
implementation web survey, the evaluation team first identified the universe of all participating 
LEAs, as well as all eligible or near-eligible LEAs. After finding that that some FSDs were 
responsible for multiple LEAs in the sampling universe, the evaluation team randomly sampled from 
among the LEAs overseen by each FSD, and included no more than two LEAs with the same 
foodservice director in the sample, to minimize burden on each individual respondent. 16 

The survey sample included each of the following respondent groups (Exhibit 2.2), which were 
defined as described in Section 2.1.2 and identified in each of the seven CEP States. 

• Eligible Participating (EP) LEAs: Survey invitations were sent to all 389 EP LEAs identified in 
the seven States. Four such LEAs were deemed ineligible (as discussed below), and of the 
remaining sample, 360 submitted surveys. 

• Eligible Non-Participating (EN) LEAs: After 31 LEAs were identified as ineligible to receive 
the survey, invitations were sent to a sample of 714 EN LEAs, and 671 completed the survey. 


14 Each year, FNS compiles a national database from the FNS-742 (VSR) reports submitted by LEAs. The 
relevant data for this evaluation include counts of identified students (approved for free meals without 
application), students approved for free and reduced-price meals by application, students in Provision 2/3 
schools not in a base year, and total enrollment. 

15 The CCD is a national database of public LEAs and schools compiled by the National Center for 
Educational Statistics ( http://nces.ed.gov/ccd/ ). SY 2010-11 CCD data were used for the evaluation. 

1(1 The restriction of the original target sample to include no more than two LEAs per FSD accounts for the 
difference between the 1,691 LEAs in the universe of near-eligible, eligible non-participating and 
participating and the 1,525 LEAs to whom the survey was initially fielded. 
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• Near-Eligible (NE) LEAs: A sample of 391 LEAs was targeted for the NE survey. 17 Sixty-six of 
these LEAs were deemed ineligible for the web survey, and the remaining 325 were asked to 
complete the survey. The final number of submitted NE surveys was 291. 

LEAs were considered ineligible for any of the web surveys if the sampling data placed the LEA in 
the wrong group (e.g., the LEA was listed as non-participating in State administrative data but 
reported participating), if the LEA was ineligible to participate in the CEP for reasons other than ISP, 
or if the FSD received an invitation to respond for three or more LEAs. The following types of LEAs 
are not eligible to participate in the CEP: Residential Child Care Institutions, LEAs that do not 
operate the NSLP, LEAs that are no longer in operation, Juvenile Detention Facilities and Hospitals. 


Exhibit 2.2: Implementation Survey Samples and Response Rates 


Survey 

Original 
Sample size 

LEAs 
Marked as 
Ineligible to 
Receive 
Survey 

Revised 
Sample size 

Number of 
Submitted 
Surveys 

Response 

Rate 

Participating (EP) 

389 

4 

385 

360 

93.5% 

Eligible non-participating 
(EN) 

745 

31 

714 

671 

94.0% 

Near-eligible (NE) 

391 

66 

325 

291 

89.5% 

Total 

1525 

101 

1,424 

1,322 

92.8% 


Three distinct web-based surveys were developed, one for each LEA group. Some questions were 
asked consistently across all three surveys, while others were purposively customized to the LEA 
group according to the data needed to address the research questions. Common topics across all 
surveys included the following: 

1. Contact information of respondent; 

2. LEA/school characteristics and environment (e.g., CEP implementation status, type of school 
meal reimbursement system previously used, staffing structure, resources available, parent 
involvement, stakeholder buy-in, and other established and/or available assistance programs); 

3. Process information (e.g., sources used to determine FRP and ISP, uses for FRP data other than 
School Breakfast Program (SBP) and the National School Lunch Program (NSLP), variability of 
ISP from year to year, determination of ISP for CEP eligibility/reimbursement, and school-level 
responsibilities regarding meal reimbursement); 

4. Communication process (e.g., process and responsibility of notifying LEAs, schools, and 
community of CEP); 

5. Decision-making process around adoption and implementation of the CEP; and 

6. Adequacy of the multiplier used to determine claiming percentages. 18 


17 To ensure the nested structure of the data collection, LEAs that were selected as comparison sites for 

primary data collection were added to the set of near-eligible LEAs if not already included in the EN or NE 
group. Thirty-four LEAs were added to the target sample. 
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As noted, some survey questions were customized to each LEA group. The following information 
was collected according to the LEA group: 

• EP web survey: This survey included questions to assess reasons for adoption of the CEP and 
perceived incentives and advantages to participation. Although this group had implemented the 
CEP, questions were also asked to assess perceived barriers to the CEP. Data were collected on 
experiences since implementation of the CEP, including: benefits and barriers encountered; 
effects of the CEP on administrative burden (e.g., reduction in burden); effects of staff 
assignments (e.g., reallocation of staff); financial impacts of the CEP (e.g., cost savings and if so, 
where; implications for Provision 2 or 3 schools); attrition of schools participating in the CEP; 
and, reasons for program continuation. 

• EN web survey: This survey included questions to assess reasons for lack of adoption of the CEP 
and perceived barriers that made the CEP unappealing (in the sense that LEAs would not want to 
participate). Questions also assessed whether there were any aspects of the CEP that seemed 
appealing, and factors that would make the CEP more appealing in the future. 

• NE web survey: This survey included questions to assess perceived incentives and barriers to the 
CEP. Data were collected on the likelihood near-eligible LEAs would participate once they 
became eligible, and what specifically would make the CEP appealing. 

2.1.4 State Child Nutrition Director Interviews 

The evaluation team conducted four rounds of telephone conversations with selected State Child 
Nutrition (CN) Directors to gather information about the implementation process. Chapter 3 presents 
results from those conversations. The first round of conversations was conducted with the three Year 
1 States that implemented the CEP in SY 2011-12 (Illinois, Kentucky, and Michigan) in January 
2012. The second round of conversations with State CN directors was conducted in the summer of 
2012 with all seven States implementing the CEP in SY 2012-13 (Year 1 States, as well as the Year 2 
States: the District of Columbia, Ohio, New York, and West Virginia). The third round of 
conversations was conducted in October 2012 with the Year 1 and Year 2 States using a structured 
interview protocol. The fourth round of conversations was conducted in summer 2013 with all States 
implementing the CEP in 2013-14 (Year 1 and Year 2 States as well as the Year 3 States: Florida, 
Georgia, Maryland, and Massachusetts), using a revised version of the structured interview protocol. 

The primary purpose of the first round of conversations in January 2012 was to obtain information to 
help guide data collection planning and instrument development. Respondents included the CN 
Directors and staff responsible for implementing the CEP in the first three States, and interviews 
addressed the following topics: 

• States’ initial contacts with LEAs about the CEP 

• LEA responses to the CEP and factors influencing adoption of the CEP 

• Existing school meals reimbursement systems and adaptations for the CEP 


18 As discussed in Chapter 1, the ISP for the school is multiplied by the statutory multiplier of 1.6 to 
determine the percentage of meals claimed at the free rate. Starting in SY 2014-15, the Secretary of 
Agriculture may establish a multiplier between 1.3 and 1.6. 
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• Availability of data for the evaluation (meal claims, enrollment, attendance, revenues, and 
administrative costs) 

• Direct certification and household application processes 

• School meals eligibility data 

• Computation of Identified Student Percentages 

The evaluation team conducted conference calls with each of the seven CEP States in the summer of 
2012. Respondents included CN directors, lead staff for the CEP, and specialists dealing with LEA 
and school data. The primary purpose of the calls was to request two kinds of data: first, lists of 
participating, eligible non-participating, and near-eligible LEAs and schools; and second, 
participation, enrollment, attendance, and revenue data. During these calls, the State staff also 
provided some general information on implementation of the CEP, particularly the response of LEAs 
to the CEP and the timing of approval of LEAs to participate for SY 2012-13. 

While earlier calls with the States yielded useful insights, the October 2012 structured telephone 
interviews with the CN Directors and other CEP staff in the seven States, together with the Summer 
2013 telephone interviews with the 11 States, represented the primary data sources on State 
implementation of the CEP for the Final Report. The topics of these interviews are listed in Exhibit 
2.3. The interview guide was tailored for each State based on prior conversations and when the States 
had begun CEP participation (Year 1 for Illinois, Kentucky, and Michigan; Year 2 for District of 
Columbia, New York, Ohio, and West Virginia; or Year 3 for Florida, Georgia, Maryland, and 
Massachusetts). 


Exhibit 2.3: State CN Agency Interview Topics 



Year 1 

Year 2 

Year 3 

Background on respondent and context 

V 

✓ 

V 

State decision to participate in CEP 

✓ 

✓ 

V 

Communication with LEAs and schools 

V 

V 

V 

Initial reaction from LEAs and schools 

s 

✓ 

s 

Eligibility data for CEP 

s 

✓ 

s 

Implementation processes 

s 

✓ 

V 

Foodservice management company contracts 

V 

V 

V 

Results and impressions 

✓ 

V 


Direct certification process 

s 

V 

s 

Communication with State education officials 

s 

V 



2.2 Impact Study Design 


The impact evaluation of the CEP was designed to assess the extent to which various outcomes for 
LEAs and schools that have implemented the CEP differ from the outcomes of those LEAs and 
schools that did not implement the CEP. While FNS was interested in examining a wide range of 
outcomes, the Impact Study design purposefully distinguished between the study’s confirmatory and 
exploratory outcomes, 19 to target evaluation resources where they were most needed and to minimize 


19 A confirmatory outcome is one that—if it shows a difference due to the intervention—can be used to judge 
the overall effectiveness or “success” of the intervention. In contrast, exploratory outcomes are those that 
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the consequences of multiple hypothesis testing. When conducting multiple hypothesis testing for 
outcomes within the same domain, the convention in social science research is to adjust by lowering 
the effective p-value at which findings reach statistical significance. There are different strategies or 
thresholds that can be imposed (Benjamini and Hochberg, 1995; Dunn, 1961). These adjustments 
effectively impose a penalty for multiple hypothesis testing. Rather than require a much larger sample 
or risk finding no statistical impacts, the evaluation team decided a priori to limit the number of 
confirmatory hypotheses to those most central to evaluating the CEP. 

The two outcomes most central to evaluating the CEP’s success are impacts on participation and 
revenues. The specific questions regarding these confirmatory outcomes were: 

• Plow did average daily participation in the NSLP change because of the CEP (compared to what 
participation would have been in the absence of the CEP)? 20 

• Plow did Federal revenues per reimbursable meal change because of the CEP (compared to 
revenues that would be generated from the traditional meal reimbursement structure)? 

As discussed below, the strongest part of the evaluation design was used to answer these confirmatory 
questions. The rest of the impact evaluation is considered exploratory, including the other Component 
1 research questions regarding participation and revenue impacts, and the research questions for 
Component 2 (administrative costs and certification errors) and Component 3 (meal quality and 
counting and claiming errors). The Implementation Study is purely descriptive, so issues of multiple 
hypothesis testing do not apply to its conclusions. 

Assessing the impact of the CEP means that one needs to know not only what has changed, but what 
has occurred in the absence of the CEP (i.e., the counterfactual). 21 For the purposes of this study, the 
effect of adopting the CEP on participation is defined as the difference between the average LEA’s 
average daily participation level in the NSLP for those LEAs that did and those that did not adopt the 
CEP. The same comparison group approach is used to estimate impacts on other outcomes (e.g., 
comparing the outcomes for LEAs that did and did not adopt the CEP). The experience of those LEAs 
that did not adopt the CEP is the best possible estimate of what participating LEAs would have 
experienced had they not chosen to participate. 


FNS, the research community, or the public would not consider central to judging the overall effectiveness 
of the CEP. Certainly, exploratory outcomes play an important role in understanding how the intervention 
might have achieved its outcomes, how outcomes vary by key subgroup, or identifying unanticipated 
consequences of an intervention. 

20 The evaluation team chose the NSLP participation as the confirmatory outcome for two reasons: the NSLP 
is larger than the SBP, and estimates of the effect of the CEP on NSLP participation are less likely to be 
biased because of the stronger design in place to assess NSLP participation impacts. The presence of the 
SBP program at a school is potentially related to a school’s choice to take up the CEP, and so disentangling 
the selection effect from the causal effect is more difficult for the SBP. Therefore, the impact on SBP 
participation is better suited for an exploratory analysis. 

21 The term “counterfactual” refers to the expected outcome if the intervention had not happened (which 
cannot be observed), as opposed to the factual outcome that is observed. 
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The strongest tests of program impacts are typically based on an experimental evaluation design that 
compares participation in a given program to what would have occurred in the absence of such 
participation, i.e., the counterfactual outcome. For the CEP evaluation, it was not feasible to 
randomize to the treatment group, due both to the timing of implementation and considerations of 
LEA autonomy. 22 Therefore, the evaluation uses the strongest feasible non-experimental design, a 
comparative interrupted time series (CITS) design, for the participation and revenue-related 
outcomes. For all other outcomes, only post-intervention measures could be obtained, and therefore 
the evaluation uses the strongest design possible without any pre-intervention measures, a posttest- 
only comparison group design. 23 Exhibit 2.4 below summarizes the analytic design for each Impact 
Study component. In the exhibit “Oi” refers to observations (“O”) in the first year covered by data 
collection (2009-10). 


Exhibit 2.4: Impact Study Outcomes and Data Collection Timing 



“01” refers to observations (“O”) in the first year covered by data collection (2009-10), and so forth. 


The Impact Study estimates program impacts by comparing mean outcomes for “treatment” LEAs 
that opted to participate in the CEP with those of similar “comparison” LEAs that did not adopt the 
CEP. The evaluation team used propensity score matching to create comparable groups of 


A regression discontinuity design was also considered but rejected as infeasible due to (1) the lack of 
sufficient LEAs with ISPs near the 40 percent eligibility threshold, and (2) the fact that LEAs could 
participate in the CEP based on having even just one eligible school. 

23 A posttest-only comparison group design compares treatment group means to comparison group means on 
outcome measures collected after the intervention has been implemented. Unlike the CITS, it does not 
include any pre-intervention measure(s) of the outcome, and therefore is less effective at ruling out some 
rival explanations for estimated impacts. 
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participating and non-participating LEAs. This method predicted LEAs’ probabilities of participation 
in the CEP to identify strong plausible matches for participating LEAs. 

Conceptually and mathematically, the difference between these treatment and comparison outcomes 
is the policy’s estimated impact. In practice, however, particularly with a non-experimental design, 
one does not simply compute the difference in outcomes to estimate program impact. Instead, the 
evaluation uses multivariate regression techniques to account for other factors that may have 
influenced selection to treatment and outcome status. 

The CITS design uses pre-implementation data points to estimate what would have happened in the 
absence of the intervention in the post-program implementation time period for both the treatment 
and comparison groups. By including a comparison group, the analysis can more accurately control 
for sources of bias that might be undetected in a simpler single-group time series design, including 
historic, economic, maturation, changing demographic, and policy influences. 

The CITS design uses data on participation and revenues for school years (SY) 2009-10 through 
2012-13, as shown in Exhibit 2.4. For these objectives, treatment group LEAs that implement the 
CEP in SY 2011-12 (Year 1) have two years of pre-implementation data and two years of post¬ 
implementation data, while those that implement in SY 2012-13 (Year 2) have three years of pre¬ 
implementation data and one year of post-implementation data. The comparison group for the CITS 
design has four years of participation and revenue data without the CEP. For the other outcomes, the 
evaluation has data only from SY 2012-13; for the treatment group, these represent post¬ 
implementation data. 

The posttest-only comparison group design allowed the study to assess the impact of the CEP on 
program integrity, staffing and administrative costs, and meal quality (Components 2 and 3). In this 
design, the treatment group is compared with a comparison group after the treatment has been 
implemented. The Posttest-Only design is the strongest feasible design for these components; it is 
certainly better than having no comparison group at all. The matched sampling methods used for 
these components sought to make the comparison group as similar as possible to the treatment group, 
reducing the selection bias introduced by the LEAs’ choices of whether to participate. However, 
unlike the analyses based on the stronger CITS design, the evaluation is unable to net out of treatment 
effects any source of bias from important underlying baseline trends. 24 

2.3 Impact Study Sampling 

This section describes the formation of the treatment and comparison groups and the baseline 
characteristics of the final sample. The Impact Study included two distinct samples: the Matched 
Administrative Data Sample for the analysis of impacts onNSLP and SBP participation and Federal 


~ 4 The administrative data, as discussed in Section 2.3, do not exclude LEAs that participate in State or local 
universal free meals programs and, as such, its comparison group represents broadly what would have 
happened in the absence of the CEP, in a world where other free universal meals programs exist. In 
contrast, the comparison group in the Component 1 PEAR data sample is one on which several exclusions 
were imposed, including the exclusion of any LEAs where free meal programs are in effect. As a result, the 
comparison is that of LEAs in which the CEP is in effect to LEAs where no universal free meal program 
exists. 


Abt Associates 


2. Study Design, Sampling, and Data | pg. 27 






Community Eligibility Provision Evaluation: Final Report 


revenues, and the Primary Data Collection Samples for Components 1, 2, and 3. These two types of 
data required somewhat different approaches to forming the treatment and comparison groups, as 
discussed in this section. As shown in Exhibit 1.1, all of the samples for the Impact Study were nested 
within the Implementation Study sample, and the Primary Data Collection Samples were successively 
nested within the Matched Administrative Data Sample. 

The remainder of this section is organized as follows. The Target Population subsection describes the 
overall population of the Impact Study and how sample eligibility requirements varied across the 
Matched Administrative Data Sample and Components 1, 2 and 3. The Administrative Data Sample 
subsection describes the propensity score modeling and matching procedures used to identify 
treatment and comparison LEAs for this sample and presents the baseline sample characteristics. The 
Primary Data Collection Sample subsection describes the propensity score matching procedures used 
to identify treatment and comparison LEAs for Components 1, 2 and 3 and then presents baseline 
sample characteristics. 

2.3.1 Target Population 

The target population of the Impact Study sample consisted of regular, public school LEAs and 
charter school LEAs in Illinois, Kentucky, Michigan, New York, Ohio and West Virginia. 25 LEAs 
serving special populations and non-public LEAs were excluded, because it was unlikely that 
appropriate matches could be identified. To ensure availability of a measure of baseline ISP, the 
sample excluded LEAs operating Provision 2 or 3 in the year prior to State implementation of the 
CEP. Similarly, LEAs missing data from key data sources essential to sampling were excluded. 26 

The samples were nested so that the LEAs selected for more intensive data collection were included 
in the less intensive components and the Matched Administrative Data Sample. The eligibility 
requirements were more restrictive for the more intensive data collection activity, because of the 
requirements for the data collection to be feasible. For example, LEAs in Component 3 had to meet 
the eligibility requirements for both Matched Administrative Data Sample and for Components 1 and 
2, because these LEAs were also included in the larger samples. In addition, LEAs in Component 3 
had to meet the Component 3-specific LEA requirements. Eligibility requirements for each sample 
are summarized in Exhibit 2.5 below, and described in detail in Appendix 2A. 


The evaluation determined that including the District of Columbia in the Impact Study was infeasible, and 
excluded these LEAs. See Appendix 2A, Section 2A.2: Sample Exclusions for a discussion of this decision. 

26 See Appendix 2A for justifications for and details of each of these sample exclusions. 
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Exhibit 2.5: Summary of Eligibility Requirements, by Impact Study Sample 


Sample 

Eligibility Requirement 

Treatment 

LEAs 

Comparison 

LEAs 

Matched 
Administrative 
Data Sample 

Regular public school LEA or public charter school LEA 

V 

V 

LEA is in Illinois, Kentucky, Michigan, New York, Ohio or 
West Virginia 

V 

•/ 

Does not exclusively serve special populations 

V 

•/ 

No schools operating Provision 2/3 in in the year prior to 
State implementation 

V 

•/ 

At least one participating school 

V 

N/A 

Component 1 

Does not offer an alternative universal free meals program 

S 

V 

No conflict with other evaluations 

S 

V 

Component 2 

Enrollment of 300 or more 

V 

V 

A school eligible for on-site data collection (i.e., not serving 
special populations) 

V 

•/ 

At least 70 percent of eligible schools participating in CEP 

V 

N/A 

Component 3 

Three schools eligible for on-site data collection that each 
represent a distinct grade range and together span 
kindergarten to 12 th grade 

s 

✓ 


For Components 1, 2 and 3, the evaluation team restricted sample inclusion to at most two LEAs per 
State that share a foodservice director (FSD). When more than two LEAs share a FSD, State-level 
respondents indicated that either: (a) the LEAs use the same foodservice management company, and 
the regional manager is the FSD for the group of LEAs, or (b) that the LEAs are charter schools run 
by a single management corporation, and the management of foodservice is centralized at the parent 
organization. Because responding to multiple surveys and interviews would represent an excessive 
burden for one FSD, the team identified groups of participating LEAs with the same FSD in each 
State and selected two LEAs to be included in the sampling frame, randomizing within a set of 
preferences that prioritized participation in the CEP, as well as size and grade coverage. Thus, any 
individual FSD was selected to participate in the evaluation for at most two LEAs within a given 
State. 

2.3.2 Matched Administrative Data Sample: Procedure and Baseline Characteristics 

The evaluation team used propensity score matching to identify participating “treatment” LEAs and 
non-participating “comparison” LEAs. The evaluation team estimated a statistical model of 
participation and generated a “propensity score.” This score was the LEA’s predicted probability of 
taking up the CEP. The propensity scores were estimated for all participating and non-participating 
LEAs in the six Impact Study States, and were then used to select the comparison sample for the 
analysis of administrative data on NSLP and SBP participation and Federal revenues. 27 The model 
included the following LEA characteristics: 

• Identified Student Percentage (ISP) 

• Percentage of students eligible for free or reduced price lunch (FRP) 


27 See Appendix 2B for a description of the model, definition of the variables, and details of the matching 
procedures. See Appendix 2A for a detailed description of the data sources and the construction of the 
sampling frame. 


Abt Associates 


2. Study Design, Sampling, and Data | pg. 29 




















Community Eligibility Provision Evaluation: Final Report 


• Proportion of NSLP meal counts provided free in the year prior to implementation in the LEA’s 
State 

• Proportion of NSLP meal counts provided at a reduced price in the year prior to implementation 
in the LEA’s State 

• Percentage of students who are English language learners (ELL) 

• Percentage of students who are in grades K-5 

• Percentage of students who are in grades 6-8 

• Whether the LEA is in an urban area (city) 

• Number of schools 

• Enrollment 

• Percentage of students who are Black 

• Percentage of students who are Elispanic 

• Percentage of schools which are Title I schools 

• Charter indicator for MI, NY and OH 

• State indicators (with KY as the excluded category) 

Exhibit 2.6 describes the baseline sample characteristics for the Matched Administrative Data 
Sample. Statistical tests demonstrated that the samples are balanced on the data elements included in 
the propensity score model. In addition, the sample is balanced on average daily participation in the 
NSLP in the year prior to State implementation of the CEP. 

The sample is not statistically balanced on average daily participation in the SBP, as the mean 
difference of 4 percentage points is statistically significant at the 5 percent level. However, the CITS 
design of this analysis uses pre-intervention measures to control for this initial difference. 
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Exhibit 2.6: Matched Administrative Data Sample Characteristics Weighted to Reflect Many to 
One Matching, LEA Level Measures 

Matched Administrative Data Sample 


Comparison 

Treatment 


Mean (SD) 

Mean (SD) 

Number of observations 

525 


286 


Weighted number of observations 

286 


286 


Enrollment 3 

3,195 

(8,795) 

4,062 

(24,235) 

Percentage distribution of students in grades K-12 

Percent in grades K-5 

50.1 

(19.8) 

51.0 

(20.2) 

Percent in grades 6-8 

23.1 

(11.2) 

23.2 

(13.3) 

Percent in grades 9-12 

24.0 

(21.2) 

22.9 

(20.2) 

Number of schools 

6.6 

(14.5) 

8.1 

(37.5) 

ISP 

52.2 

(18.2) 

53.9 

(17.6) 

Percent students free/reduced lunch 

72.2 

(17.9) 

74.0 

(16.8) 

Percent Title 1 schools 

83.3 

(23.8) 

85.0 

(21.4) 

Urban LEA (%) 

31.3 

(46.4) 

33.6 

(47.3) 

All charter schools (%) 

32.3 

(46.8) 

33.2 

(47.2) 

Percent English Language Learners 

3.7 

(10.9) 

3.7 

(10.6) 

Percent students Black 

30.3 

(35.4) 

33.2 

(36.3) 

Percent students Hispanic/Latino 

7.8 

(14.1) 

7.1 

(12.2) 

Average daily participation (%) lunch 

70.7 

(12.8) 

71.9 

(12.3) 

Average daily participation (%) breakfast 0 

42.1 

(21.3)*“ 

47.1 

(24.8)“* 


SD=Standard deviation. 

Hypothesis testing performed using t-tests for all outcomes except for grade distribution. Grade distribution 
outcomes jointly tested using MANOVA. The means and standard deviations reflect the weights assigned by the 
propensity score matching algorithm, which selected up to 5 comparisons for each treatment LEA. 
a The large SD for enrollment reflects the inclusion of some very large LEAs in the treatment group, including 
New York City and Chicago. 

b Observed for 517 comparison LEAs (weighted number of observations 279.6) and for 284 treatment LEAs. 
***=p<.01, **=p<.05, *=p<.10. 

2.3.3 Primary Data Collection Sample: Matching Procedure and Baseline Characteristics 

The evaluation team used similar propensity score matching to identify matched pairs of 
participating, “treatment” LEAs and non-participating, “comparison” LEAs within each State for the 
primary data collection (Components 1, 2, and 3). The propensity scores for the primary data 
collection sample were based on a model using the following LEA characteristics: 

• Identified Student Percentage (ISP) 

• Percentage of students eligible for free or reduced price lunch (FRP) 

• Percentage of students who are English language learners (ELL) 

• Percentage of students who are in grades K-5 

• Percentage of students who are in grades 6-8 

• Whether the LEA is in an urban area (city) 

• Number of schools 
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• Enrollment 

• Charter indicator 

Additional details describing the construction of the primary data collection sample are available in 
Appendix 2C. 

Exhibit 2.7 describes the baseline sample characteristics for the Component 1, 2 and 3 LEAs. All 
LEAs with sufficient data to include in at least one analysis are included in the appropriate 
components. Hence, the number of observations listed in the table is the maximum number of LEAs 
included in analyses for a particular component. 

Statistical tests were conducted to determine the extent of balance between the treatment and 
comparison groups. The more balanced such samples are, the more plausible it is that the samples are 
similar in not only observed but also unobserved characteristics. As shown in Exhibit 2.7, these tests 
demonstrated that the samples were balanced on measures of enrollment, grade distribution, number 
of schools, FRP percentage, Title I status, whether the LEA was located in a city, charter status, and 
demographic characteristics. 28 In addition, the samples were balanced on average daily participation 
(ADP) in the NSLP in the year prior to State implementation of the CEP. 

Although the Component 1 and 2 samples were balanced on many characteristics, they were not 
balanced on ISP or on ADP in the School Breakfast Program in the year prior to State implementation 
of the CEP. Both the 5 percentage point ISP difference between Component 1 treatment and 
comparison LEAs and the 7 percentage ISP point difference between Component 2 treatment and 
comparison LEAs are statistically significant at the 5 percent level. The differences between 
treatment and comparison LEAs in ADP in the SBP were of larger magnitude—8 percentage points 
for Component 1 and 12 percentage points for Component 2—and significantly different at the 5 
percent level. The Component 3 differences were of similar magnitude to the Component 1 
differences, and have similar standard deviations. The differences for Component 3 were not 
statistically significant, although the sample size was too small to detect the difference. 

For the PEAR survey analyses, which used the CITS design, the pre-intervention measures for the 
treatment group controlled for pre-intervention differences between the treatment and comparison 
groups that affect the outcome of interest. However, the analyses conducted using the Component 2 
and 3 samples relied on a posttest-only comparison group design. Hence, the initial differences 
between the groups are important to bear in mind when interpreting the impact findings from analyses 
of the Component 2 and 3 samples. In this report, the presentation of each impact analysis includes 
the results of balance tests between the samples and the variables used to control for differences in 
sample characteristics in order to separate out the effects of the CEP. 


28 For consistency of representation, these variables were defined as they were for the Matched 
Administrative Data Sample Characteristics table: using the data sources and definitions from the 
Administrative data propensity score model—see Appendix 2B—and the analysis of participation—see 
Chapter 5. 
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Exhibit 2.7: Components 1,2 & 3 Sample Characteristics, LEA-Level Measures 




Component 1 



Component 2 



Component 3 



Comparison 

Treatment 

Comparison 

Treatment 

Comparison 

Treatment 


Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Number of observations 

108 


144 


51 


52 


25 


27 


Enrollment 

3,022 

(4,604) 

2,957 

(4,909) 

3,854 

(5,752) 

4,060 

(6,955) 

5,546 

(7,539) 

6,803 

(8,806) 

Enrollment 1-499 (%) 

25.0 


33.3 


17.6 


21.2 


0.0 


3.7 


Enrollment 500-2,499 (%) 

41.7 


31.3 


45.1 


36.5 


48.0 


29.6 


Enrollment 2,500-4,999 (%) 

17.6 


18.8 


13.7 


25.0 


20.0 


33.3 


Enrollment 5,000+ (%) 

15.7 


16.7 


23.5 


17.3 


32.0 


33.3 


Percentage distribution of students in grades K-12 

Percent in grades K-5 

50.1 

(16.7) 

50.0 

(18.1) 

49.7 

(17.1) 

51.0 

(13.9) 

44.0 

(4.4) 

44.3 

(3.4) 

Percent in grades 6-8 

22.8 

(6.6) 

22.0 

(9.1) 

21.0 

(6.3) 

23.5 

(7.9) 

21.9 

(1.7) 

21.8 

(17) 

Percent in grades 9-12 

23.6 

(17.4) 

24.7 

(20.7) 

25.9 

(17.6) 

21.7 

(13.5) 

30.1 

(5.5) 

29.5 

(3.3) 

Number of schools 

6.1 

(7.3) 

6.6 

(9.2) 

7.5 

(8.8) 

8.9 

(12.8) 

10.7 

(11.1) 

14.4 

(15.8) 

1 school (%) 

24.3 


30.5 


17.6 


17.6 


0.0 


0.0 


2-5 schools (%) 

42.1 


29.8 


37.3 


37.3 


36.0 


29.6 


6-14 schools (%) 

26.2 


27.0 


35.3 


27.5 


44.0 


37.0 


15+ schools (%) 

7.5 


9.9 


9.8 


13.7 


20.0 


25.9 


ISP 

46.1 

(16.2)** 

51.0 

(17.7)** 

47.9 

(16.2)** 

55.2 

(16.5)** 

44.5 

(11.1) 

49.6 

(12.3) 

Percent students free/reduced lunch 

67.9 

(16.1) 

70.7 

(16.5) 

70.6 

(15.2) 

74.3 

(14.6) 

66.2 

(14.2) 

70.3 

(12.7) 

Percent Title 1 schools 

83.7 

(22.0) 

85.4 

(20.4) 

86.4 

(21.2) 

87.1 

(15.1) 

79.1 

(25.6) 

80.4 

(14.4) 

Urban LEA (%) 

23.1 


30.6 


27.5 


30.8 


20.0 


33.3 


All charter schools (%) 

19.4 


27.1 


15.7 


19.2 


0.0 


3.7 


Percent English Language Learners 

2.8 

(6.3) 

2.3 

(6.4) 

3.1 

(5.1) 

3.4 

(6.9) 

3.2 

(4.4) 

4.5 

(8.2) 

Percent students Black 

26.1 

(33.2) 

29.9 

(36.5) 

27.0 

(31.3) 

33.5 

(35.9) 

15.5 

(19.5) 

24.2 

(28.5) 

Percent students Hispanic/Latino 

7.0 

(11.7) 

6.4 

(10.8) 

8.0 

(11.6) 

8.9 

(15.0) 

7.6 

(10.5) 

10.5 

(17.8) 

Average Daily Participation (%) Lunch 

70.4 

(10.6) 

69.8 

(13.6) 

68.5 

(11.1) 

70.8 

(12.8) 

65.6 

(10.5) 

67.2 

(15.1) 

Average Daily Participation (%) Breakfast 3 

37.1 

(25.3)** 

44.8 

(27.6)** 

36.8 

(20.6)** 

49.2 

(28.3)** 

32.3 

(16.5) 

39.8 

(16.0) 


Hypothesis testing performed using t-tests for all outcomes except for grade distribution. Grade distribution outcomes jointly tested using MANOVA. 


***=p<.01, **=p<.05, *=p<.10. 

a Component 1: observed for 105 comparison LEAs and 143 treatment LEAs. Observed for all LEAs in Component 2 and Component 3. 
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2.4 Impact Study Data Collection 

As described below, the Impact Study included collection of administrative records and primary data. 
The primary data collection effort included three components aligned with the research objectives. 
Program participation and revenues data were collected through the combination of the Component 1 
PEAR survey and State administrative data. Component 2 collected data on administrative costs and 
certification errors by conducting in-person interviews and record abstractions. Component 3 
collected data on meal quality and meal counting/claiming errors through on-site observations. For 
Component 1, data were collected at the LEA level. For Components 2 and 3, data were collected at 
the LEA and the school levels, as shown in Exhibit 1.1. 

Exhibit 2.8 maps research questions to the data sources described below. 
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Exhibit 2.8: Research Objectives and Questions, Role in Evaluation, and Designs 


Research 

Objective 

Research Questions 

Role in 
Evaluation 

Design 

Sample 

Data Source 

Impact on 

How did average daily participation in the NSLP and 

NSLP: 

Comparative 

Matched 

Administrative 

SBP and 

SBP change because of the CEP compared to what 

Confirmatory 

Interrupted Time 

Administrative 

Data 

NSLP 

participation 

participation would have been in the absence of the 
CEP? 

SBP: 

Exploratory 

Series (CITS) 

Data Sample 



To what extent (if any) did the impacts of the CEP on 
program participation: 

Reflect changes in the number of actual participants, 
frequency of participation, or both for NSLP, SBP, and 
total meals? 

Vary by grade level (elementary, middle, and high 
school) for NSLP, SBP, and total meals? 

Exploratory 

Posttest-Only 

Comparison 

Group, and self- 
assessment by 
LEAs 

Component 1 

PEAR Survey 

Impact on 
foodservice 

revenues 

How did Federal revenues change because of the CEP 
compared to revenues that would be generated from 
traditional meal reimbursement structure? 

Confirmatory 

CITS 

Matched 

Administrative 
Data Sample 

Administrative 

Data (Federal 
Revenue: NSLP 
and SBP) 


How did revenues from non-Federal sources, including 
State funds, local funds, student payments for 
reimbursable meals, and payments for competitive 
foods and adult foods, change because of the CEP 
compared to revenues that would be generated from 
traditional meal reimbursement structure? 

How do revenues compare to revenues that would be 
generated under other special assistance alternatives 
(Provision 2 or 3)? 

How much local funding is needed compared to these 
other alternatives? 

Exploratory 

CITS 

Component 1 

Administrative 

Data and PEAR 
Survey Data 
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Research 


Role in 




Objective 

Research Questions 

Evaluation 

Design 

Sample 

Data Source 


Impact on 
School 

Breakfast 

Program 

How did the CEP impact the School Breakfast Program 
in terms of whether schools offer breakfast and the type 
of breakfast service offered? 

Exploratory 

Pretest-Posttest 
Comparison Group 
design 

supplemented with 
self-reported 
reason for change 
(for presence) 

Posttest-Only 
Comparison Group 
(for type) 

Component 1 
(PEAR Survey) 

Component 2 
(Pre-visit School 
Questionnaire) 

PEAR Survey Data 
(Presence) 

Pre-visit School 

Questionnaire 

(Type) 


To what extent did the CEP affect the foods served at 

breakfast, specifically: 

• Whether the identical breakfast is served to all 
students (or more generally, whether the CEP affects 
the amount of breakfast food choices for students)? 

• Whether and to what extent a la carte and 
competitive foods are offered during breakfast? 

How much time are students allowed to eat breakfast? 

Exploratory 

Posttest-Only 
Comparison Group 

Component 2 

Pre-visit School 
Questionnaire 

Impact on 

administration 

(staff 

allocation and 
costs, non¬ 
program 
needs for 
eligibility 
data, direct 
certification) 

To what extent did the administrative burden of 
operating the NSLP and SBP change under the CEP? 

What is the difference in administrative costs to the LEA 
associated with the CEP compared to the costs in 
absence of the CEP? 

Exploratory 

Posttest-Only 
Comparison Group 

Component 1 

Component 2 

PEAR Survey 
(Change) 

Administrative 

Cost Interviews 

Impact on 
foodservice 

costs 

To what extent did the costs of producing reimbursable 
meals change under the CEP? 

Exploratory 

Posttest-Only 
Comparison Group 

Component 1 

PEAR Survey 
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Research 

Objective 

Research Questions 

Role in 
Evaluation 

Design 

Sample 

Data Source 

Impact on 

program 

integrity 

How did the CEP impact program integrity, including: 
administrative errors associated with the certification 
process, and errors in meal counting by cashiers and in 
computing claims for reimbursement? 

Exploratory 

Posttest-Only 
Comparison Group 

Component 2 

Component 3 

Component 3 

Certification 

Record Review 

Cashier 

Observations 

Meal Counting and 
Claiming 

Impact on 

nutritional 

quality 

(meeting 

standards, 

variety, 

competitive 

foods) 

To what extent, if any, did the nutritional profile of 
school meals offered change under the CEP compared 
to what would have happened in the absence of the 
CEP? 

To what extent, if any, did the variety of food choices 
offered to students in school meals change? 

To what extent, if any, did compliance with USDA-FNS 
nutrition standards change? 

Exploratory 

Posttest-Only 
Comparison Group 

Component 3 

Menu Survey 
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2.4.1 Administrative Data on Participation, Enrollment, Attendance and Revenue 

The administrative data on participation, enrollment, attendance and revenue were obtained from the 
six Impact Study States (Illinois, Kentucky, Michigan, New York, Ohio and West Virginia) to 
address research questions regarding impacts on participation and foodservice revenues. All six States 
provided data on NSLP and SBP meal counts, number of operating days, and Federal payments from 
NSLP, SBP and the afterschool snack program. Most States provided additional requested data 
elements, as summarized below in Exhibit 2.9. The State- provided data include observations for all 
SFAs in the State, and cover school years 2009-10 to 2011-12 and the first half of 2012-13. 


Exhibit 2.9: Meal Count and Revenue Administrative Data Provided by each State 


Data Elements 

IL 

KY 

Ml 

NY 

OH 

WV 

1. Number of NSLP lunches reimbursed 

Yes 

Yes 

Yes 

Yes 0 

Yes 

Yes 

2. Number of SBP breakfasts reimbursed 

Yes 

Yes 

Yes 

Yes 0 

Yes 

Yes 

3. Number of operating days 

Yes 

Yes 

Yes 

Yes 0 

Yes 

Yes 

4. Student enrollment (preferably as of October 31) 

Yes 

Yes 

Yes 

Yes 0 

Yes 

Yes 

5. Average daily student attendance 

Yes 

Yes c 

No 

No 

No 

Yes 

6. School foodservice revenues: 

a. Federal payment from National School Lunch 
Program 

Yes 

Yes 

Yes 

Yes 

Yes® 

Yes 

b. Federal payment from School Breakfast Program 

Yes 

Yes 

Yes 

Yes 

Yes® 

Yes 

c. Federal payment from Afterschool Snack Program 

Yes 

Yes 

Yes 

Yes 

Yes® 

Yes 

d. Other Federal payment (e.g., FFVP, Special Milk, 
CACFP, equipment grants, etc.) 

Yes ab 

Yes b 

No 

Yes 

Yes b 

Yes f 

e. Value of commodities received 

Yes ab 

No 

No 

No 

Yes ab 

No 

f. State payments 

Yes 

Yes b 

No 

Yes 

Yes ab 

No 

g. Local funds (other than payment for catering) 

No 

N/A 

No 

No 

No 

No 

h. Student payment for reimbursable meals 

No 

Yes® 

No 

No 

No 

No 

i. Non-reimbursable foods (e.g., a la carte, vending 
machines) 

No 

Yes b 

No 

No 

No 

No 

j. Other revenue from local sources not listed above 
(e.g., catering, special events) 

No 

Yes b 

No 

No 

No 

No 


a Provided yearly data. 

b Provided data only for 2009-10 to 2011-12 school year. 

c Provided average daily attendance for 2009-10 to 2011-12 school year and average daily participation for 
2012-13 school year. 

d Provided monthly data only for Oct, Nov and Dec for 2009-10 to 2011-12 school year. 
e Provided semiannual data instead of monthly data for 2012-13 school year. 
f Provided yearly data only for 2010-11 and 2011-12 school year. 

2.4.2 Participation, Enrollment, Attendance and Revenue (PEAR) Survey 

The LEA PEAR Survey was a web-based survey fielded to 330 LEAs (164 treatment and 166 
comparison) from February to May 2013. The PEAR Survey uses the Component 1 sample to address 
research questions regarding impacts on participation, foodservice revenues, and presence of the SBP. 
The PEAR survey asked LEA foodservice director about those data elements on the participation, 
enrollment, attendance and revenue not provided by the State as part of the Administrative Data 
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Collection. 29 To investigate whether the CEP affects the number of participants or their rate of 
participation and the impacts by grade level, the LEA survey asked foodservice directors for their 
perceptions about these facets of the impacts. A total of 57 LEAs were screened out of the PEAR 
survey due to a change in CEP participation status (17 comparison) or due to participation in an 
alternative universal free meals program (16 treatment and 24 comparison). A total of 246 LEAs 
completed all of the priority PEAR survey items, for a response rate of 90.1 percent. 30 

2.4.3 Administrative Cost Interviews 

The evaluation measured the impacts of the CEP on the costs of administrative activities and the staff 
hours for these activities by measuring and comparing costs and hours for the samples of participating 
and non-participating LEAs in the matched comparison sample for Component 2 (which also 
included the certification record reviews described below). Administrative cost interviews were 
conducted for 52 treatment and 48 comparison LEAs. Within each selected LEA, up to three schools 
were sampled; fewer schools were sampled in LEAs with only one or two schools. Admini s trative 
cost data were collected in a total of 247 schools, 123 treatment and 124 comparison. For LEAs that 
did not implement the CEP universally, schools were sampled from those that were using the CEP. A 
total of 5 LEAs (2 treatment and 3 comparison) were found to be ineligible for the Impact Study after 
completion of data collection. The final analysis sample included 123 treatment schools and 129 
comparison schools within 52 treatment and 48 comparison LEAs. 

Costs and hours were measured for the following activities that were expected to be affected by the 
CEP: 

• Processing FRP meals applications 

• Direct certification, including ISP calculation, and certification without applications from other 
sources (such as foster child lists) 

• Verifying FRP meals applications 

• Managing student payment accounts and payment systems 

• Preparing and submitting claims for reimbursement 

For each activity, data collectors obtained estimates of staff time and labor costs through interviews 
with the LEA foodservice director, school cafeteria managers, and (where necessary) school 
principals or other school personnel. Data collection, and the resulting comparisons, focused on 
salaries and other labor costs (rather than other direct costs such as office supplies and capital 
expenditures) because labor represents by far the largest component of administrative costs. In 


29 The survey was adapted for each State to omit requests for information which the State had agreed to 
provide as part of the Administrative Data on Participation, Enrollment, Attendance and Revenue. 

30 The following categories of PEAR survey items were designated as priority items for follow-up: 
respondent information, information on LEA and number of schools participating in CEP, respondent 
perceptions of changes in average daily participation in the NSLP and SBP, and respondent perceptions of 
changes in food, labor and costs in the NSLP and SBP. Respondents that completed the priority items were 
considered to have completed the survey. 
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addition to these quantitative measures of time and costs, staff in participating LEAs provided 
qualitative information on factors including: 

• The extent to which they have reallocated or reduced staff or other costs since they implemented 
the CEP, including positions eliminated and staff who have been reassigned. 

• Any changes made to meal planning as a result of implementing the CEP. Participating LEAs 
may simplify menus because they are now serving more meals, or they may be able to provide 
additional choices (or healthier choices) due to the changes. 

This qualitative information was intended for use in describing the response of LEAs to changes (if 
any) in staff hours and costs for the specified activities. 

2.4.4 Certification Record Reviews 

To obtain data on administrative certification errors, data collectors reviewed eligibility documents 
maintained by LEAs, including approved and denied applications for FRP meals, direct certification 
lists, and other related documents. For this data collection, student records were sampled. In CEP 
schools, the sample consisted of identified students (approved for free meals without application). In 
non-CEP schools, the sample included identified students, students approved for FRP meals by 
application, and students whose applications were denied. Data collectors completed forms to abstract 
data from eligibility lists, FRP meals applications, and other documentation supporting LEAs’ 
eligibility determinations. These data were analyzed to determine whether approved applications were 
completed according to program rules and had valid documentation, and whether the eligibility 
determinations (including approvals and denials) were correct. 

Certification record data were collected from the Component 2 sample in 103 LEAs and 252 schools 
selected for the administrative cost interviews. Data collection included 114 records of identified 
students in treatment schools, and 114 records of certified students and 36 records of denied 
applicants in comparison schools. 

2.4.5 Menu Survey 

The Menu Survey was conducted with the Component 3 sample of participating and non-participating 
LEAs. Data were collected for a total of 162 schools (81 treatment and 81 comparison). Among the 
sampled schools, 6 comparison schools were found to be ineligible for the Impact Study after 
completion of data collection. The final sample included 81 treatment schools and 75 comparison 
schools within 27 treatment and 25 comparison LEAs. 

The Menu Survey was completed by school foodservice staff from their production records. It 
obtained data on all foods offered in reimbursable meals for the target week (up to 5 days), including 
detailed food descriptions, serving sizes, planned number of servings, manufacturer information, and 
recipes for school-prepared foods. These data were coded and processed by the evaluation team using 
specialized menu analysis software to determine the nutritional profile of lunch and breakfast as 
offered, the number of choices in each food group, and the extent of compliance with USDA 
standards for NSLP and SBP meals. 

2.4.6 Cashier Observations 

In the Component 3 sample of schools that participated in the Menu Survey, data collectors recorded 
the reimbursable foods included in meals (taken by a random sample of students for both breakfast 


pg. 40 | 2. Study Design, Sampling, and Data 


Abt Associates 




Community Eligibility Provision Evaluation: Final Report 


and lunch) and the corresponding determination of meal classification (reimbursable or not) made by 
the cashier for each meal observed. The purpose was to obtain a measure of the rate of cashier error. 
Data were collected for each meal offered by the school on one day of the target week at each of the 
sampled schools in the LEA. Treatment schools were required to offer the SBP, but in comparison 
schools, breakfast was observed only in schools that had the SBP. 

Data were collected for 27 treatment and 27 comparison LEAs; within these LEAs, completed data 
were obtained for a total of 162 schools (81 treatment and 81 comparison). The final sample includes 
81 treatment schools and 75 comparison schools within 27 treatment and 25 comparison LEAs. 31 In 
each school, a total of 40 breakfast and 60 lunch observations were conducted. 

2.4.7 Meal Counting and Claiming 

The meal counting and claiming data were collected to investigate errors in counting, recording, and 
claiming reimbursement for school meals. Errors could occur at several points in the counting and 
claiming process—at the individual school level when consolidating the different cashier totals in a 
school or when consolidating school totals into a single LEA claim. Or, errors could occur when 
reported information was incorrectly transcribed onto the necessary form—for example, when 
entering school-reported information into a State web-based claim system. The meal counting and 
claiming forms were designed to provide a basis for estimating the percentage of reimbursement 
dollars paid in error. 

Data collectors obtained meal counts at the school and LEA levels to permit three comparisons that 
might identify meal counting and claiming errors: 

1. Meal counts reported by the school to the LEA for one sampled day versus supporting records at 
the school (such as tally sheets or cash register tapes). 

2. Meal counts reported by the school to the LEA for the sampled day versus the record of those 
counts at the LEA central office. 

3. Meal counts reported by schools to the LEA for a sampled claiming week or month versus the 
meal counts reported by the LEA to the State for that period. Depending on the claiming 
procedure, these counts were for the sampled schools or all schools in the LEA. 


31 Two LEAs, both comparison, were found to be ineligible for the Impact Study after completion of data 
collection. 
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3. Implementation of the Community Eligibility Provision 


The experiences of the 11 participating States in their first, second, or third year of implementation, 
and of both participating and non-participating LEAs, provide important insights into key questions 
about implementing the Community Eligibility Provision (CEP). The specific research questions 
about aspects of implementation among the 11 States include: 

• What factors influenced the decision to adopt the CEP at the State level? 

• What was the process of deciding about CEP participation at the LEA level? 

• What communication methods did States use to notify and identify LEAs, and how did LEAs 
communicate with schools and community members? 

• What are the benefits and barriers to CEP participation, as perceived by LEAs? 

• What State systems had to be adapted to accommodate the CEP? What processes did States have 
to adjust when implementing the CEP? 

• What changes to meal variety, staffing, processes and costs did LEAs report? 

• What are the results and lessons learned from CEP implementation at the State and LEA levels? 

Additionally, FNS sought to learn about potential ramifications of the CEP for educational programs 
that currently use FRP meals application data, which are not available for schools using the CEP. 

This chapter presents answers to these questions, based on four different data sources: 1) interviews 
in Summer 2013 with the 11 State CN directors and other State staff members and earlier 
conversations with Year 1 and 2 States, 2) telephone surveys of 51 State Title I directors in 
Spring/Summer 2013, 3) web surveys of LEA foodservice directors in Winter/Spring 2013, and 4) 
administrative cost interviews conducted with LEA foodservice directors in CEP districts in 
Winter/Spring 2013. The LEA data were collected only in the Year 1 and Year 2 States, and the 
interviews with the four Year 3 States reflect only early implementation prior to the 2013-14 school 
year. Each of the four sources is described in Chapter 2. 

This chapter first focuses on State-level decisions to participate in the CEP, and then communications 
and decision-making of the LEAs, schools, and community. Next, the perceived benefits and barriers 
to participation are described. Third, some of the processes and methods the State and participating 
LEAs and schools adjusted to during CEP implementation are detailed. The chapter concludes with 
findings about implementation, including successes and benefits, challenges and barriers, and 
implications for educational programs that use FRP meals eligibility data. 

3.1 Decision-Making and Notification of LEAs, Schools and the Community 

Within the State CN Agencies, the first step in CEP implementation was the decision to apply for the 
CEP. In SY 2011-12, FNS selected three States based on criteria about which States were most likely 
to have high numbers of eligible LEAs and schools. In each of the next two years, all remaining 
States received a memo about the availability of the CEP; five applied for SY 2012-13 (four were 
selected) and six applied for SY 2013-14 (four were selected), bringing the total to 11 participating 
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States. This section describes State participation decisions, LEA participation decisions, and 
communications about the CEP among States, LEAs, schools, and communities. 

3.1.1 State Participation Decision 

Several factors influenced the States’ decisions to participate in the CEP. The factors most commonly 
cited by Year 1 and 2 States were (1) the opportunity to provide free, nutritious breakfasts and 
lunches to more students, and the (2) potential to decrease the administrative burden of application 
processing and meal counting for LEAs. Other factors identified by one or two States were additional 
reimbursement to LEAs and the elimination of the verification process. Year 3 States reported that 
family financial need in the State and the streamlined process of CEP were factors in their decision. 

3.1.2 LEA Participation Decision Process 

A variety of LEA stakeholders were involved in the decision about whether to adopt the CEP. The 
foodservice director and superintendent were involved in a substantial majority of participating LEAs 
(79 percent and 78 percent respectively; Exhibit 3.1). Other common stakeholders involved in 
participating LEAs’ decision-making processes included other LEA administrators (61 percent), the 
School Board (60 percent), and principals (46 percent). For eligible non-participating LEAs, the most 
frequently mentioned stakeholder was the foodservice director (62 percent). 


Exhibit 3.1: Percentage of LEAs Indicating Different Stakeholders Were Moderately or Very 
Involved in the Decision Whether to Adopt the CEP 


Stakeholders 

Participating LEAs 

Eligible Non- 
Participating LEAs 

Students 

6.4 

1.2 

Parents 

11.9 

2.1 

Teachers 

12.2 

2.2 

Principals 

46.4 

15.8 

Cafeteria managers 

34.2 

14.5 

Foodservice director 

79.2 

61.9 

Superintendent 

77.5 

48.9 

Other LEA administrators 

61.1 

37.1 

School board 

59.7 

16.1 

State Department of Education 

44.7 

15.4 

Governor 

4.2 

0.5 

Other elected officials 

3.3 

1.0 

Community-based organizations 

4.2 

0.8 

Number of responding LEAs 

360 

671 


Note: Multiple responses were allowed so percentages sum to more than 100. 
Source: Surveys of Participating and Eligible Non-Participating LEAs. 


The stakeholder most frequently making LEAs’ final decision about participation in the CEP was the 
School Board (37 percent), followed by the superintendent (28 percent) (Exhibit 3.2). For eligible 
non-participating LEAs, most often the superintendent (23 percent) and foodservice director (20 
percent) were the final decision-makers. (As indicated in Exhibit 3.2, about one-fifth of the eligible 
non-participating LEAs were not sure who made the final decision; the primary respondent for these 
surveys was the SFA director, who was not necessarily involved in the final decision.) 
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Exhibit 3.2: Percentage of LEAs Indicating Which Stakeholders Made the Final Decision about 
CEP Participation 

Stakeholders 

Participating LEAs 

Eligible Non- 
Participating LEAs 

State Department of Education 

1.7 

3.6 

LEA superintendent 

28.1 

23.4 

School board 

37.1 

6.8 

LEA foodservice director 

12.8 

20.2 

Other LEA administrator 

8.1 

11.2 

Other 

6.4 

13.8 

Don’t know 

5.9 

21.0 

Number of responding LEAs 

359 

662 


Source: Surveys of Participating and Eligible Non-Participating LEAs. 


Once the decision was made to participate in the CEP, almost all of the eligible schools participated. 
In the participating LEAs, on average, 93 percent of the eligible schools participated; 97 percent of 
eligible schools participated in LEAs where all schools were eligible and 79 percent participated in 
LEAs where some but not all schools were eligible. (See Chapter 4 for further information on school- 
level participation.) 

3.1.3 State Communications with LEAs 

Implementation of the CEP required initial communications between FNS, States, LEAs, and schools. 
In all States, these communications began when FNS publicized the opportunity for States to apply to 
implement the CEP. 32 This announcement resulted in communications between States and LEAs 
about potential interest. Once States were selected, FNS required them to notify eligible and 
potentially eligible LEAs. The Year 1 States first conducted this process in 2011, and the Year 2 
States first conducted the process in 2012. All three groups of States notified LEAs in 2013 in 
preparation for SY 2013-14. 

Most eligible non-participating LEAs in the first seven States reported having been informed about 
the CEP, but less than 80 percent reported this in three States. Near-eligible LEAs were generally less 
likely to report having been informed (less than 80 percent in five of the seven States; Exhibit 3.3). 


Exhibit 3.3: Percentages of Non-Participating LEAs that Reported Being Informed About the 
CEP by the State 



District of 
Columbia 

Illinois 

Kentucky 

Michigan 

New 

York 

Ohio 

West 

Virginia 

Eligible non¬ 
participating LEAs 

83.3 

66.67 

96.7 

74.9 

83.3 

62.0 

100.0 

Near-eligible LEAs 

100.0 

58.2 

84.6 

61.3 

64.0 

30.9 

(a) 


Percentages in table are based on 671 eligible non-participating, and 291 near-eligible non-participating 
respondents, (a) Data are suppressed because West Virginia had only one near-eligible LEA. 

Source: Surveys of Eligible Non-Participating and Near-Eligible LEAs. 


2 As specified in Chapter 1, the Year 1 States (Illinois, Michigan, and Kentucky) first implemented the CEP 
in SY 2011-12, Year 2 States (District of Columbia, New York, Ohio, and West Virginia) first 
implemented the CEP in SY 2012-13, and Year 3 States (Florida, Georgia, Massachusetts and Maryland) 
implemented the CEP in SY 2013-14. 
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Among LEAs that reported contacts, they were most commonly contacted by State officials about the 
CEP. Some LEAs reported that they had been informed by the USDA/FNS (Exhibits 3.4a and 3.4b), 
although it is unclear in these cases if the information provided by FNS had actually been forwarded 
to the LEAs by the States. 


Exhibit 3.4a: Percentages of LEAs that Were Informed by a USDA/FNS or State Official about 
the CEP by Type of LEA 


Participating 

Eligible Non- 
Participating 

Near-Eligible 

LEAs 

LEAs 

LEAs 


USDA/FNS 9.2 10.6 5.8 


State official 70.6 51.0 33.0 

Number of responding LEAs 360 671 291 

Source: Surveys of Participating, Eligible Non-Participating, and Near-Eligible LEAs. 

Exhibit 3.4b: Percentages of LEAs that Were Informed by a USDA/FNS or State Official about 
the CEP by State 



District of 
Columbia 

Illinois 

Kentucky 

Michigan 

New 

York 

Ohio 

West 

Virginia 

USDA/FNS 

2.9 

7.4 

15.6 

10.5 

14.2 

5.3 

0.0 

State official 

76.5 

45.4 

68.8 

52.1 

46.2 

40.6 

92.6 


Percentages in Exhibits 3.4a and 3.4b are based on 360 eligible and participating LEAs, 671 eligible non¬ 
participating LEAs, and 291 near-eligible non-participating respondents. The percentages reported in Exhibits 
3.4a and 3.4b are based on all responding LEAs, not on the subset of LEAs informed about the existence of the 
CEP, so the percentages do not sum to 100%. 

Source: Surveys of Participating, Eligible Non-Participating, and Near-Eligible LEAs. 

States used numerous modes to inform LEAs about the CEP, including emails, presentations and 
meetings, training workshops, webinars, one-on-one phone calls, site visits, newsletters, and Web 
postings (according to State CN director interviews). Ten of the 11 States reported engaging in 
multiple strategies of communication depending on the timing of the communication process and the 
level of interest among LEAs. From the perspective of LEAs, email was the most co mm on form of 
communication about the CEP, with in-person meetings and postal mail as the next co mm on methods 
(Exhibit 3.5). Word-of-mouth among LEAs was also common; eight of the 11 States, including all of 
the Year 1 States, reported that they encouraged LEAs to talk with one another about the CEP and 
facilitated this communication. For example, one State reported that they left time for an open 
discussion among LEAs at the end of a webinar. One Year 3 State reported having facilitated 
communication between one of its large LEAs and a large LEA in a Year 1 State that was similar in 
demographics, so the LEA could learn more about implementation of the CEP. 

The majority (nine) of States reported providing estimators, online calculators, or worksheets 
(developed by USD A or the State) for LEAs to use in determining whether they were eligible to 
participate. These tools enabled LEAs to enter data about numbers of identified students into a 
database and receive notification of eligibility status. 

States varied in their perceptions about the most effective strategies for communicating with LEAs 
about the CEP. States commented that emails and webinars were cost-effective and successful form s 
of communication. Three States noted that providing individualized technical assistance to LEAs was 
very important, and one State reported that “a high level of technical assistance was a key to their 
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success in communicating effectively” with LEAs. Several States described unique strategies that 
they perceived to be effective, such as conducting a poster session at a State meeting, and obtaining 
the help of a community-based organization to promote interest through the media. While most States 
directed their communications to LEA foodservice directors, one State’s Superintendent of Education 
promoted the CEP directly to LEA superintendents to encourage participation. The majority (eight) of 
States did not report any challenges in their notification and communication process with LEAs. 
Challenges mentioned by the remaining three States included LEA concerns about not having FRP 
lunch data, not having individual school ISP data, and not having enough time for communication 
with LEAs. 


Exhibit 3.5: Percentages of LEAs Reporting Methods of Communicating Information about the 
CEP From States to LEAs 



Participating 

LEAs 

Eligible Non- 
Participating 
LEAs 

Near-Eligible 

LEAs 

In-person meeting 

32.8 

24.6 

11.7 

In-person training 

18.1 

12.5 

5.2 

Letter/mail 

24.7 

18.8 

12.4 

Email 

54.4 

49.8 

32.7 

Phone call 

8.1 

1.9 

2.1 

Webinar 

15.3 

14.2 

8.3 

Press release 

11.1 

8.5 

4.5 

Other LEAs 

8.9 

9.4 

5.8 

Other 

4.4 

2.5 

5.2 

Don’t know 

7.5 

2.8 

1.7 

Number of responding LEAs 

360 

671 

291 


Multiple responses were allowed so percentages sum to more than 100. 

Source: Surveys of Participating, Eligible Non-Participating, and Near-Eligible LEAs. 


States also differed in terms of which types of LEAs they targeted for CEP communications. At a 
minimum, States were required to notify eligible LEAs with overall ISPs of 40 percent or more as 
well as potentially eligible LEAs with ISPs of 30 percent to 40 percent. All 11 States reported that 
they initially informed all LEAs about the CEP, regardless of ISP. Subsequent communication 
appeared to be directed toward eligible LEAs based on their respective ISPs. 13 One State sent 
targeted communications to LEAs participating in Provision 2 or 3 as well as LEAs with 
accountability or performance issues for whom the CEP might be particularly beneficial. Another 
State targeted its pre-proposal and follow-up communications to a subgroup of the largest LEAs that 
appeared to meet or exceed eligibility criteria based on FRP student data. States differed in their 
capacity to identify eligible schools. Six States indicated that they identified individual eligible 
schools on the basis of direct certification data, and four indicated that they relied on LEAs to identify 
eligible schools. 14 (This issue is discussed in the section on benefits and challenges to CEP 
implementation.) 


Three States explicitly stated in interviews that they targeted outreach to eligible and potentially eligible 
LEAs. Others provided a more general description of their outreach. 

34 For one Year 3 State, the availability of school-level ISPs was unknown. 
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3.1.4 LEA Communications with Schools and Community Members 

Almost one-quarter of the eligible non-participating LEAs (24 percent) and almost half of the near- 
eligible participating LEAs (48 percent) reported that they had not been informed about the CEP 
(Exhibit 3.6). LEAs in Year 1 States were less often uninformed about the CEP, compared with LEAs 
in Year 2 States (23 versus 29 percent for eligible non-participating LEAs, and 39 versus 57 percent 
for near-eligible LEAs). Only 28 percent of all eligible non-participating LEAs and 24 percent of all 
near-eligible LEAs in turn informed their schools. (Eligible non-participating LEAs were required to 
notify their eligible schools about the CEP, but near-eligible LEAs did not have any notification 
requirements.) 


Exhibit 3.6: Percentages of LEAs that Informed Their Schools about the CEP 


Stakeholders 

Eligible Non- 
Participating LEAs 

Near-Eligible LEAs 

LEA was not informed about CEP 

24.4 

48.1 

LEAs informed about CEP 

LEA informed schools 

28.2 

23.7 

LEA did not inform schools 

32.6 

19.6 

Don’t know if LEA informed schools 

13.3 

7.9 

LEAs that did not respond to question 

1.5 

0.7 

Percent (total) 

100.0 

100.0 

Number of responding LEAs 

671 

291 


Source: Surveys of Eligible Non-Participating and Near-Eligible LEAs. 


The most common methods used by LEAs to communicate with their schools about the CEP were in- 
person meetings and presentations (over 50 percent of each LEA type), followed by email notification 
(over 30 percent of each LEA type). About one-third (34 percent) of eligible participating LEAs used 
postal mail to inform schools (Exhibit 3.7). 


Exhibit 3.7: Percentage of LEAs by Method of Communication with Schools about the CEP 


Method of Communication 

Participating 

LEAs 

Eligible Non- 
Participating 
LEAs 

Near-Eligible 

LEAs 

In-person meeting/presentation 

61.0 

55.0 

68.1 

In-person training 

16.9 

11.1 

8.7 

Letter/mail 

34.1 

9.0 

2.9 

Email 

38.4 

38.1 

30.4 

School website 

19.8 

0.5 

0.0 

Phone call 

14.0 

11.6 

8.7 

Webinar 

2.6 

0.0 

0.0 

Press release 

18.1 

1.1 

0.0 

Other 

3.4 

7.9 

14.5 

Don’t know 

6.3 

2.7 

1.5 

Number of responding LEAs 

349 

189 

69 


Multiple responses were allowed so percentages sum to more than 100. 

Source: Surveys of Participating, Eligible Non-Participating, and Near-Eligible LEAs. 


Almost all (94 percent) eligible participating LEAs reported that they had informed their communities 
in some way about the CEP, but as might be expected, the eligible non-participating and near-eligible 
LEAs only did so infrequently (less than 10 percent of each type). LEAs communicated with their 
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respective communities in a variety of ways. The most co mm on methods for participating LEAs were 
a notice or letter sent home with students (59 percent) and postal mail (52 percent) (Exhibit 3.8). 


Exhibit 3.8: Percentage of LEAs by Method of Communication with the Community about the 
CEP 


Method of Communication 

Participating LEAs 

In-person meeting/presentation 

21.5 

Letter/mail 

52.4 

Email 

9.7 

School website 

43.5 

Phone call 

7.1 

Notice/letter sent home with students 

58.5 

Local newspaper 

38.2 

Other 

5.9 

Don’t know 

6.5 

Number of responding LEAs 

349 


Only 38 eligible non-participating LEAs and 13 near-eligible LEAs responded to this question. Multiple responses 
were allowed so percentages sum to more than 100. 

Source: Surveys of Participating, Eligible Non-Participating, and Near-Eligible LEAs. 


The foodservice director (FSD) was the most likely type of LEA official responsible for initially 
informing schools about the CEP (over 40 percent of each LEA type) (Exhibit 3.9). 

Exhibit 3.9: Percentage of LEAs by Who was Responsible for Initially Communicating with 
Schools 


LEA Official 

Participating 

LEAs 

Eligible Non- 
Participating 
LEAs 

Near-Eligible 

LEAs 

Superintendent 

21.0 

15.1 

30.9 

Foodservice director 

46.1 

56.8 

39.7 

Foodservice contract management co. 

2.9 

2.7 

0.0 

Other LEA administrator 

11.7 

11.4 

16.2 

Other 

10.2 

10.8 

13.2 

Don’t know 

8.2 

3.2 

0.0 

Number of responding LEAs 

343 

185 

68 


Source: Surveys of Participating, Eligible Non-Participating, and Near-Eligible LEAs. 


In eligible participating LEAs, the FSDs were also most likely to inform the community (38 percent) 
followed by the Superintendent (20 percent) (Exhibit 3.10). 
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Exhibit 3.10: Percentage of LEAs by Who Was Responsible for Informing the Community 

About the CEP 

LEA Official 

Participating LEAs 

Superintendent 

19.5 

Foodservice director 

38.3 

Foodservice contract management co. 

0.9 

Other LEA administrator 

9.0 

Principals of individual schools 

16.2 

Other 

9.6 

Don’t know 

6.6 

Number of responding LEAs 

334 

Only 38 eligible non-participating LEAs and 13 near-eligible LEAs responded to this question. 

Source: Survey of Participating LEAs. 

3.2 Benefits and Barriers to CEP Participation 



The State CN director interviews and LEA surveys provide perspectives on two important aspects of 
CEP participation decisions. First, States and LEAs described the benefits and barriers to 
participation. Second, States and LEAs shared their perceptions of the factors influencing LEA 
participation decisions. Each aspect is described below. 

3.2.1 Perceived Benefits 

State interviews indicate that six of seven Year 1 and 2 States perceived LEAs’ initial reactions to the 
CEP as generally positive, albeit with some concerns. According to these States, LEAs understood the 
potential benefits of participation in the CEP, including: free breakfast and lunch for all children 
attending participating schools, elimination of the burden of the free and reduced price meal 
application for families and foodservice staff, and simplification of the meal counting process. The 
seventh State reported mixed reactions: eligible LEAs were positive while non-eligible LEAs were 
not, particularly those with ISPs just below the 40 percent cut-off for eligibility. Four States 
mentioned LEA nutrition staff as particularly positive about the CEP, and two specifically reported 
that LEAs had received positive feedback from parents. 

Two of the four Year 3 States noted that one of the biggest benefits is being able to provide free 
meals to all students without any stigma. Two Year 3 States also commented on the efficiency of the 
CEP process, and one State observed that not having to process free and reduced price meal 
applications allowed the LEAs to “focus on more important issues like whether students are getting 
nutritious meals.” 

Survey data indicate that a majority of participating LEAs (81 percent) characterized the possibility of 
relief for families under financial burden as an important potential benefit from participation in the 
CEP. Nearly three-fourths of participating LEAs reported increasing school meal participation (73 
percent) as a potential benefit; other benefits included decreasing stigma for children in need (55 
percent), and improved academic performance (51 percent). A significantly lower proportion of 
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eligible non-participating and near-eligible LEAs indicated that these benefits were important 
(Exhibit 3. II). 35 


Exhibit 3.11: Percentage of LEAs Rating Possible Benefits of Participation as Moderately or 
Very Important 


Possible Benefit 

Participating 

LEAs 

Eligible Non- 
Participating 
LEAs 

Near-Eligible 

LEAs 

Increased revenue*** 

42.9 

13.6 

30.3 

Decreased costs*** 

20.6 

8.2 

21.4 

Decreased administrative burden*** 

47.3 

24.6 

40.0 

Decreased stigma for students in need*** 

54.9 

22.7 

32.1 

Improved academic performance*** 

51.0 

15.7 

30.9 

Increased school meal participation*** 

73.0 

34.8 

48.1 

Improved nutritional quality of meals*** 

33.6 

7.6 

17.9 

Relief for families under financial 
burden*** 

80.5 

36.6 

51.2 

Number of responding LEAs 

359 

671 

291 


Percentages in table are based on 359 participating LEA respondents, 671 eligible and non-participating 
respondents, and 291 near-eligible respondents. Multiple responses were allowed so percentages sum to more 
than 100. 

For each possible benefit, a chi square test (1 df) was performed to compare endorsement rates of participating 
and eligible non-participating LEAs. 

***=p<.01, **=p<.05, *=p<.10. 

Source: Surveys of Participating, Eligible Non-Participating, and Near-Eligible LEAs. 

Furthermore, over half (55 percent) of eligible participating LEAs reported that participation in the 
CEP actually reduced administrative burden for their LEAs. In particular, the three most commonly 
reported types of personnel whose burden was reduced were: LEA foodservice administrative staff 
(45 percent), school foodservice workers (28 percent), and school administrators (25 percent) (Exhibit 
3.12). 


Exhibit 3.12: Percentages of LEAs Citing Personnel Whose Burden Has Been Reduced by 
Implementing the CEP 


Personnel 

Participating LEAs 

School administrators 

25.3 

LEA foodservice administrative staff 

45.0 

Other LEA administrative staff 

12.8 

Cafeteria managers 

17.8 

School foodservice workers 

28.3 

Number of responding LEAs 

360 


Source: Survey of Participating LEAs. 


35 Significance tests were performed for comparisons of participating and eligible non-participating LEAs on 
questions about key attitudes likely to affect the decision to participate. The purpose was to test whether 
these key attitudes were different between these two groups that had the option to participate. The near- 
eligible LEAs were not included in these tests because they did not have the option to participate. 
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Among LEAs that reported reduced burden, more than half (54 percent) indicated that due to 
participation in the CEP, staff members were able to spend more time overseeing foodservice 
operations and managing other administrative tasks (53 percent) (Exhibit 3.13). Over one-third (36 
percent) reported that staff spent more time on planning nutrition or meal services. 


Exhibit 3.13: Percentages of Participating LEAs Reporting Activities Staff Were Able to Spend 
More Time On Due to Participating in the CEP (Among Those Reporting Any Reduced Burden) 


Activity 

Participating LEAs 

None 

14.2 

Conducting professional development or training 

16.2 

Conducting other administrative tasks 

53.3 

Implementing new school breakfast program 

6.6 

Planning nutrition/meal services 

36.0 

Overseeing foodservice operations 

53.8 

Developing recipes 

13.2 

Researching purchasing options for food and supplies 

18.3 

Developing nutrition education materials/activities 

15.2 

Assessing student nutritional needs and/or food preferences 

29.4 

Other 

4.1 

Number of responding LEAs 

197 


Multiple responses were allowed so percentages sum to more than 100. 
Source: Survey of Participating LEAs. 


The most important factors for participating LEAs in their decision to participate were the rate of 
reimbursement for school meals (55 percent) and their respective community poverty level (48 
percent) (Exhibit 3.14). These two factors were identified by significantly fewer eligible non¬ 
participating LEAs as decision-making factors (41 and 9 percent, respectively). For eligible non¬ 
participating LEAs, the potential financial impact was the most commonly reported factor in deciding 
not to participate in the CEP (59 percent). 36 A sizable majority of eligible non-participating and near- 
eligible LEAs agreed that the CEP would be well-received by school staff and students (86 percent 
and 83 percent, respectively) and would result in increased meal participation that would benefit 
students (83 percent and 79 percent, respectively) (Exhibit 3.15). 

State CN directors agreed with some of the factors affecting LEA participation as reported by LEAs, 
and offered some additional perspectives. Three States reported that support from LEA FSDs or 
administrators was an important factor in an LEA’s decision to participate. Two States perceived that 
community poverty level affected LEAs’ decisions to participate. Other factors affecting LEAs’ 
participation, according to States, were perceived benefits for children, high proportion of FRP 
lunches served, and pressure from neighboring communities. 


36 The mean ISP for 356 eligible participating LEAs was 46.8 percent, and the mean ISP for 659 eligible non¬ 
participating LEAs was 36.9 percent, a difference of 9.9 percent. This is statistically significant, p<.0001. 
While potential financial impact was the most commonly reported factor in eligible non-participating 
LEAs’ decisions about whether to participate, whether this factor disinclined these LEAs to participate is 
not known. For instance, some eligible non-participating LEAs may have been attracted by CEP because of 
a potential positive financial impact, yet may have decided against participation for other reasons. 
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Exhibit 3.14: Percentage of LEAs Endorsing Factors as Among the Three Most Important in 
Deciding Whether or Not to Participate 

Factor 

Participating LEAs 

Eligible Non- 
Participating LEAs 

Poverty of the community*** 

48.1 

9.2 

ISP or rate of reimbursement for school meals*** 

55.3 

40.5 

Staffing needs** 

4.2 

8.5 

Financial impact*** 

46.4 

58.6 

Rate of participation in school meals programs*** 

25.8 

11.5 

Logistics or ease of implementation 

13.3 

10.4 

Considerations around schools being labeled as low income 

1.9 

2.8 

Considerations around students being labeled as low income 

2.8 

3.0 

Availability of the CEP for the long term*** 

29.4 

14.0 

Eligibility of individual schools or entire LEA to participate 

20.3 

21.9 

Number of responding LEAs 

360 

671 


Multiple responses were allowed so percentages sum to more than 100. 

For each possible benefit, a chi square test (1 df) was performed to compare endorsement rates of participating 
and eligible non-participating LEAs. 

***=p<.01, **=p<.05, *=p<.10. 

Source: Surveys of Participating and Eligible Non-Participating LEAs. 
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Exhibit 3.15: Percentages of Non-Participating LEAs Agreeing with Beliefs About the CEP 



Strongly Agree 

Somewhat Agree 

Somewhat Disagree 

Strongly Disagree 


Eligible Non- 


Eligible Non- 


Eligible Non- 


Eligible Non- 


1 believe that the CEP would... 

Participating 

LEAs 

Near-Eligible 

LEAs 

Participating 

LEAs 

Near-Eligible 

LEAs 

Participating 

LEAs 

Near-Eligible 

LEAs 

Participating 

LEAs 

Near-Eligible 

LEAs 

Be more costly to implement 
than what is currently in place. 

35.2 

29.3 

35.7 

41.5 

21.0 

20.7 

8.1 

8.5 

Be well-received by school staff 
and students. 

43.5 

33.7 

42.4 

49.1 

10.9 

14.7 

3.2 

2.6 

Be confusing to implement. 

12.5 

13.7 

37.8 

47.4 

31.2 

28.9 

18.5 

10.0 

Result in increased meal 
participation that would be 
overwhelming to the staff. 

7.5 

10.7 

21.1 

25.5 

42.3 

42.1 

29.1 

21.8 

Result in increased meal 
participation that would benefit 
students (access to more foods, 
healthy foods). 

41.7 

38.4 

41.7 

41.0 

11.5 

16.2 

5.1 

4.4 

Create issues for determining 
eligibility for other assistance 
programs due to lack of free and 
reduced price data. 

28.2 

26.3 

39.5 

49.6 

21.4 

17.0 

11.0 

7.0 

Result in a lot more work to 
serve breakfast. 

7.9 

16.2 

21.5 

26.9 

33.4 

35.4 

37.1 

21.4 

Increase plate waste. 

10.0 

18.9 

30.8 

34.4 

36.7 

36.3 

22.4 

10.4 


Percentages in table are based on 629 eligible non-participating LEAs and 273 near-eligible (non-participating) LEAs. Multiple responses were allowed so percentages sum to more 
than 100. 

Source: Surveys of Eligible Non-Participating and Near-Eligible LEAs. 
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More than half of participating LEAs indicated that total reimbursements under the CEP were greater 
than conventional meal claims in prior years (Exhibit 3.16). Over 80 percent of LEAs in Kentucky 
and New York reported receiving more reimbursement. In contrast, the proportion of LEAs reporting 
having received less reimbursement under the CEP was 20 percent or less in each State. . 

Exhibit 3.16: Percentages of Participating LEAs Indicating How Reimbursements under the 
CEP in SY 2012-13 Compare to Reimbursements Received in Prior Years 


Reimbursements in SY 2012-13 Compared to Prior Years 



Much More 

Moderately 

More 

Same 

Amount 

Moderately 

Less 

Much Less 

District of Columbia 

10.0 

45.0 

25.0 

15.0 

5.0 

Illinois 

12.0 

44.0 

28.0 

16.0 

0 

Kentucky 

7.8 

76.5 

11.8 

2.0 

0 

Michigan 

16.2 

58.1 

16.2 

9.5 

0 

New York 

18.2 

70.5 

9.1 

2.3 

0 

Ohio 

12.7 

52.7 

21.8 

9.1 

0 

West Virginia 

11.4 

54.3 

22.9 

11.4 

0 

All States 

13.3 

58.3 

18.3 

8.9 

0.3 


Percentages in table are based on 360 eligible and participating LEAs.1 LEA from Kentucky and 2 LEAs from 
Ohio did not respond to this question. 

Source: Survey of Participating LEAs. 


3.2.2 Perceived Barriers 

Almost half of participating LEAs (45 percent) indicated that concern about how the CEP might 
affect funding for educational programs was an important barrier to participation (Exhibit 3.17). 
(Other programs still required free and reduced price student data; this issue is discussed below and in 
Section 3.4). LEAs also were often concerned about how much reimbursement would be received (41 
percent). The most common concern reported by eligible non-participating LEAs (53 percent) was the 
financial viability of the CEP; the amount of reimbursement was a related concern (41 percent). 

State CN directors reiterated the LEA concerns identified by surveys, and provided additional insights 
about some of the questions and issues faced by LEAs. 

• All 11 States mentioned that LEAs had questions about or were concerned about the CEP’s 

potential effect on both Federal (i.e., Title I and E-rate) 37 , and State funding, given that individual 
student free and reduced price meal eligibility data would no longer be available. (This issue is 
discussed further in the final section of this chapter.) Ten of 11 States reported that LEAs had 
specific questions and concerns about Title I funding, and several States commented that this 
concern kept several large LEAs in their State from participating in the CEP. In another State, 
LEAs were reported to be most concerned with the lack of individual household income data, as 
it affected compensatory education funding provided by the State to LEAs. Four States 
commented that it would have been useful to have had more specific Federal guidance about the 
potential implications for allocation of Federal and State funds. 


37 


Title I provides Federal educational assistance to high-poverty schools. E-rate provides assistance for 
telecommunications and Internet access to high-poverty schools and other institutions. 
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Exhibit 3.17: Percentage of LEAs Rating Possible Barriers to Participation as Moderately or 
Very Important 

Possible Barrier 

Participating LEAs 

Eligible Non- 
Participating LEAs 

Near-Eligible LEAs 

CEP not financially viable*** 

29.1 

53.0 

41.4 

Concern about how much 

reimbursement would be received 

41.2 

40.5 

55.3 

Concern about how the CEP will affect 
funding for educational programs*** 

45.1 

29.7 

34.7 

Not enough time to implement the CEP 

11.7 

13.0 

14.1 

Concern about participating schools 
being treated differently than other 
schools*** 

7.2 

13.9 

8.6 

Participating LEAs may be viewed as 
poor 

4.8 

6.9 

7.2 

Difficulty establishing a school 
breakfast program 

0.8 

1.2 

3.8 

Community not supportive 

1.9 

1.6 

5.5 

Key LEA or school officials not 
supportive 

3.9 

3.7 

4.5 

Number of responding LEAs 

360 

671 

291 


Percentages in table are based on 360 participating LEA respondents, 671 eligible and non-participating 
respondents, and 291 near-eligible respondents. Multiple responses were allowed so percentages sum to more 
than 100. 

For each possible barrier, a chi square test (1 df) was performed to compare endorsement rates of participating 
and eligible non-participating LEAs. 

***=p<.01, **=p<.05, *=p<.10. 

Source: Surveys of Participating, Eligible Non-Participating, and Near-Eligible LEAs. 

• Another initial concern (cited by three States in the 2012 interviews) was that not enough time 
was provided for LEAs to understand the CEP, make sure that the CEP was financially viable, 
and obtain local approval for participation. These three States indicated that the time constraints 
limited LEA participation. Similar concerns were raised by two of the four Year 3 States in 2013 
interviews. Several Year 1 and 2 States commented on how their previous experience enabled 
them to begin the application process earlier in 2012-13. 

• Two States noted that some LEAs were concerned about fairness in the implementation of the 
CEP. Since LEAs could group schools within the LEA to achieve the 40 percent ISP, some LEAs 
were concerned that the CEP would provide free meals to all students in schools with relatively 
few students who would qualify for free meals under conventional procedures. 

• Finally, another fairness issue arose for some LEAs when CEP schools offered free meals to all 
students, while other schools in the same LEA had to raise meal prices to meet the new paid 
meals equity regulation. According to States, parents perceived this situation as having the non¬ 
participating schools subsidize the participating schools. 

• Five States reported at least one question raised by LEAs for which they had to seek guidance 
from FNS, including questions about mixed LEAs in which not all schools were participating in 
CEP. For example, one LEA asked how to handle meals for students who visit other schools if 
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one school was implementing CEP and the other was not. Another question had to do with 
schools that merged if one school was implementing CEP and one was not. 

When eligible non-participating LEAs were asked about changes that would make the CEP more 
appealing, about two-thirds (65 percent) named an increased reimbursement rate (Exhibit 3.18). The 
second most popular change endorsed by eligible non-participating LEAs was to make the CEP 
available LEA-wide regardless of individual schools’ ISPs (45 percent). One-third (33 percent) of 
eligible non-participating LEAs indicated that they would find the CEP more attractive if alternative 
methods (other than via FRP data) could be found to qualify, and about one-quarter (26 percent) 
wanted greater lead time prior to implementation. LEAs were asked whether they would be willing or 
unwilling to participate in the CEP using different (lower) multipliers that ranged from 1.5-1.0; less 
than 10 percent of each LEA type indicated willingness to participate, and a majority indicated that 
they did not know (Exhibit 3.19). 


Exhibit 3.18: Percentage of Eligible Non-Participating LEAs Indicating Changes that Would 
Make the CEP More Appealing 



Number of responding LEAs 671 


Multiple responses were allowed so percentages sum to more than 100. 
Source: Survey of Eligible Non-Participating LEAs. 


Exhibit 3.19: Percentages of LEAs Willing to Participate in the CEP at Lower Multipliers 



Participating LEAs 

Eligible Non-Participating 
LEAs 

Near-Eligible LEAs 

Multiplier 

Willing 

Un¬ 

willing 

Don’t 

Know 

Willing 

Un¬ 

willing 

Don’t 

Know 

Willing 

Un¬ 

willing 

Don’t 

Know 

1.0 

1.7 

31.5 

66.8 

0.3 

45.7 

54.0 

0.3 

41.1 

58.6 

1.1 

1.7 

31.4 

66.9 

0.2 

45.6 

54.2 

0.7 

41.0 

58.3 

1.2 

2.0 

30.9 

67.1 

0.2 

45.8 

54.0 

0.7 

41.0 

58.3 

1.3 

3.3 

29.8 

66.9 

0.5 

45.3 

54.2 

1.0 

40.7 

58.3 

1.4 

6.7 

28.7 

64.6 

0.9 

45.1 

54.0 

1.4 

40.3 

58.3 

1.5 

8.6 

27.5 

63.9 

0.8 

45.0 

54.2 

2.8 

39.8 

57.4 

Number of 
responding 

LEAs 


369 



670 



291 



Each row represents a separate question. 

Source: Surveys of Participating, Eligible Non-Participating, and Near-Eligible LEAs. 


3.2.3 State Perceptions of Types of LEAs Most Likely to Participate 

LEAs and schools with higher ISPs were more likely to participate in the CEP, according to State 
officials (and as confirmed by the analysis of LEA participation presented in Chapter 4). A key 
threshold cited by States was that LEAs with ISPs of 62.5 percent or more could claim all meals at 
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the free rate, making the CEP a “no-brainer” in the words of one State respondent. Other perceived 
influences on CEP participation included size, type of LEA, buy-in from LEA leadership, and use of 
Provisions 2 and 3 (which, like the CEP, eliminate applications for FRP meals and provide free meals 
to all students). (See Chapter 4 for detailed tables of LEA characteristics by State.) The reasons for 
these factors being influential are discussed below. 

• Size: According to two States, in smaller LEAs the processes for getting approval for 
implementation were both simpler and quicker, or because the potential for reduced 
administrative burden was more important. On the other hand, one State mentioned that large 
urban LEAs had spent years refining their direct certification methods to increase their numbers 
of directly certified students, thus boosting their ISPs and the desirability of the CEP. 

• Type of LEA: According to two States, many charter schools have high concentrations of low- 
income students and thus high ISPs and claiming percentages. These two States also explained 
that charter schools generally did not face the problem confronted by regular public school 
districts, particularly larger ones: that the CEP would result in reallocating funds away from the 
schools with the highest concentrations of students in poverty (as discussed later in this section). 

• LEA administrator support: Four States emphasized the importance of support from LEA 
administrators, and in some cases school leadership, for the CEP. They observed that LEA 
leadership had to be comfortable with and supportive of an initiative that involved change and 
uncertainties. 

• Provisions 2 and 3: For schools using these provisions, the CEP was particularly appealing, 
according to the States with these schools, for several reasons. First, these schools had already 
decided to offer free meals for all students. Second, they had relatively high ISPs and often 
qualified to have all meals reimbursed at the free rate (more than under Provision 2/3). 38 Third, 
two States observed that taking FRP meals applications is more difficult for Provision 2/3 schools 
because they do not conduct the process annually, so eliminating the process entirely under the 
CEP is particularly appealing. Finally, in one State, these schools were already collecting 
alternate household income data and did not report the same concerns about Federal funding as 
LEAs using the traditional reimbursement method. 

3.3 State-Level and LEA-Level Implementation 

State-level implementation for the CEP included changes to meal reimbursement systems and direct 
certification. At the LEA-level, changes to meals, staffing, and costs were identified. 

3.3.1 Adaptation of Meal Reimbursement Systems 

Eight States adapted their meal reimbursement systems to accommodate the CEP, and of the 
remaining three, two States are in the process of developing a new system that will incorporate CEP 
in SY 2014-15, and the other State reported that it did not need to modify its system. In two of the 
States without a modified system for CEP, LEAs used a spreadsheet provided by the States to 


" Qualifying for the CEP could be challenging for schools using Provision 2 or 3 that are not in a base year, 
and therefore may not have completed the direct certification process. However, in States that provide 
direct certification results at the school level, ISP data are available for these schools. 
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calculate counts of free and paid meals. One of these two States indicated that it provided the 
spreadsheet because it did not have time to modify its claiming system. States did not report any 
specific CEP-related challenges with this process. 

All 11 States currently have an automated on-line meal claim system for NSLP and SBP. Nine States 
pre-loaded the CEP claiming percentages for participating LEAs and schools into their meal claiming 
systems to automate the computation by CEP LEAs of meals to be claimed at the free and paid 
rates. 39 The automated systems also helped to ensure that checks are in place for LEAs to use the 
correct claiming percentage. As described above, two of the other three States provided spreadsheets 
for LEAs to make these computations. Five of the 11 States reported minor issues with the accuracy 
of claims from a small number of LEAs, but no major issues with State verification of LEA data were 
reported. 

One potential issue for implementation of the CEP is that meal claims could become more 
complicated if the LEA submits a consolidated claim for all of its schools but only some of the 
schools are operating under the CEP (a “mixed” LEA). Elowever, no States reported any 
complications with claims from mixed LEAs. In the nine States with systems programmed to apply 
approved claiming percentage for CEP schools, claims were entered and processed in the usual 
manner for non-CEP schools. 

3.3.2 Direct Certification Process 

All States reported that they were encouraging CEP LEAs to continue annual direct certification. 
Three States require ongoing direct certification for LEAs without 100 percent free reimbursement, 
and one State requires all LEAs to continue. All States mentioned two main reasons for the 
importance of ongoing direct certification: 1) the possibility of increasing ISPs, and 2) the need for 
direct certification data for allocation and reporting for other funding streams. 

Six States reported positive changes to the direct certification process during SY 2012-13 that 
allowed them to better identify directly certified children through system enhancements and therefore 
increase the ISPs of LEAs and schools. Examples of enhancements include flagging certain students 
(e.g. SNAP, TANF, foster children) in automated systems, use of a student identifier number to find 
students on direct certification lists, and structuring the system so that all students in a household 
could be identified through one student’s record. Two other States reported more frequent extraction 
of data over the course of the year. Several other States also mentioned that they were part of the 
Medicaid direct certification pilot, which may increase ISPs. 

Starting in 2013, a new reporting form (FNS 834) requires States to provide a count of students who 
are matched to SNAP records. This information is needed for all LEAs, including those that 
implement the CEP and are not required to conduct direct certification. All but two States planned to 
obtain this infonnation through identifying SNAP students based on direct certification data. Of the 
remaining two, one planned to obtain the needed information from another report, and the other was 
formulating plans at the time of the interview. As noted above, only six States indicated that they had 


39 As explained in Chapter 1, the claiming percentage for the CEP is the proportion of meals reimbursed with 
Federal funds at the free meals rate (currently 1.6 times the ISP). LEAs count the total meals served and 
then submit claims for free and paid meals based on the claiming percentage. 
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direct certification data at the school level for all LEAs, so four or five States would have to obtain 
the information for the new FNS report from LEAs. 

3.3.3 Changes to Meal Variety, Staffing, Process & Cost 

On-site interviews with LEA foodservice directors indicated that about equal numbers reported 
having made some or no changes to the variety of meals, staffing, and other processes as a result of 
the CEP. The majority of LEA foodservice directors (36 of 54) reported no change to the variety of 
foods offered as a result of the CEP. Almost one-fifth (9) reported that they had increased the variety. 
Examples of changes included using more fresh fruits instead of canned/frozen fruits, and introducing 
new food options to students. Given new meal patterns that were being implemented at the same time, 
the extent to which CEP affected variety in these LEAs is unclear. 

Over half of FSDs (30) reported no changes to the types of food served, and one-third (18) indicated 
they had made changes. Offering grab-and-go breakfast was the most common change reported. Over 
half of FSDs (31) indicated the amount of fresh fruits and vegetables had not increased because of the 
CEP. One-third (18) reported that the amount did increase. Over half of directors (31) indicated that 
they had not made changes to the serving process in order to serve more quickly. About one-third 
(20) reported that they had; a co mm on example was increasing the number of serving lines. 

Most FSDs (38) noted they had not changed foodservice administrative staffing as a result of the 
CEP. Less than one-fifth (8) reduced staff. Food production staffing had generally remained stable; 
over half (30) of respondents had made no changes, and almost one-third (17) increased staff hours. 

In terms of data processing, such as the meal counting system, over half (30) of FSDs reported no 
changes due to the CEP; less than half (22) indicated that data processing had changed. Some 
examples of data processing changes included not having to track individual names, and instead 
simply using a less time-consuming tally sheet. Finally, the most common response about whether the 
CEP affected LEAs’ capacity to break even was that the CEP made this easier (24) and almost one- 
fifth (10) said this had stayed the same. 

3.4 Results and Lessons Learned from CEP Implementation 

The Year 1 States—Illinois, Kentucky, and Michigan—succeeded in implementing the CEP in SY 
2011-12 and SY 2012-13. The Year 2 States—the District of Columbia, New York, Ohio, and West 
Virginia—did so in SY 2012-13. The Year 3 States - Florida, Georgia, Maryland and Massachusetts 
- will begin implementation in SY 2013-14. 40 

3.4.1 Benefits and Challenges for State and Local Implementation of the CEP 

States described several important benefits of CEP implementation. When asked about the biggest 
benefit of CEP, six States reported that school breakfast and lunch participation had increased. Four 
States reported that CEP eliminated the stigma for students who received free and reduced price 
lunches. Another benefit mentioned by four States was the ability for LEA staff to focus more on 
meal content and patterns than on administrative activities. 


40 The evaluation team was collecting information on LEA participation in the CEP for SY 2013-14 at the 
time of this report. A supplement to this report will provide this information. 
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States also reported having to deal with several implementation-related challenges. The largest 
challenge for CEP States was trying to determine how CEP will affect other funding sources and 
programs that use FRP data, in particular Title I. In addressing this challenge, States worked with 
LEAs to figure out how CEP would affect them financially. (This topic is described in further detail 
below). 

As noted above, another important challenge for the Year 1 and 2 States was the timing of CEP 
implementation. For example, the Year 2 States were notified in May 2012 that they had been chosen 
for the CEP. Thus, they had four months to identify and notify eligible and potentially eligible LEAs, 
respond to LEA questions, encourage LEAs to apply, process LEA applications to participate, and 
modify their meal claims systems. States had to complete these steps at the same time that they were 
processing LEA applications to participate in the NSLP and SBP, and implementing and providing 
training on other program changes. Furthermore, the States were asked to provide LEA and school 
data for the evaluation. Six Year 1 and 2 States reported that time constraints impeded their efforts to 
recruit LEAs to take up the CEP during the first two years of implementation. Only one Year 3 State 
reported time constraints during SY 2012-13. 

A related challenge for States was the need to develop the process for LEAs to apply for the CEP. 

Five States mentioned that a clearinghouse or manual for CEP implementation from FNS would have 
better facilitated implementation. Year 1 States had frequent communications with one another to 
adapt to this initial challenge, including sharing State-developed forms and procedures (such as 
worksheets for LEAs to use during the initial application process). One Year 1 State reported that it 
was in the process of developing its own CEP policy and procedures manual; this State emphasized 
the need for the Year 2 States to be aware of implementation process lessons learned by Year 1 
States. FNS conducted periodic conference calls among the CEP States to share problems and lessons 
learned; members of the evaluation team participated in two of these calls. All of the Year 3 States 
indicated that initial implementation had been relatively smooth because of lessons and processes 
learned from Year 1 and 2 States. 

At least four States, and possibly five, lacked school-level direct certification data at the State level. 
These States had to obtain data from LEAs as part of their applications for the CEP to identify 
eligible schools and, therefore, were not able to identify LEAs with eligible schools if they did not 
apply for the CEP. As discussed above, the lack of school-level direct certification data in these 
States also posed a challenge for completing the FNS-834 report. 

All Year 1 and 2 States reported receiving very positive feedback from participating LEAs and 
schools about the CEP. States reported few implementation challenges for LEAs and/or schools. One 
unanticipated issue within mixed districts was parental confusion about why one child could eat for 
free and the other could not. Another State described the lack of local infrastructure to change the 
system and reporting, particularly in small LEAs. 

3.4.2 Benefits and Barriers to LEA Participation in the CEP 

As described earlier, there appeared to be a clear link between the anticipated benefits of the CEP and 
LEAs’ participation decisions. Participating LEAs cited relief for families, increased participation, 
reduced stigma, and improved school performance most often as the expected benefits. Eligible non¬ 
participating LEAs had less positive expectations about these benefits. Over half of participating 
LEAs reported increased reimbursements and reduced administrative burden, but these appeared to be 
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less important motivators. The States perceived all of these factors as potential benefits when they 
chose to implement the CEP. 

Concerns about the financial impacts of the CEP appeared to be the largest barriers to participation 
among eligible non-participating and near-eligible LEAs. The majority of eligible non-participating 
LEAs reported that the CEP was not financially viable for them, yet described the CEP as potentially 
more appealing if (a) reimbursement were higher or (b) if the CEP were available in all schools. 

Other barriers included concerns about equity issues when schools are treated differently (especially 
when prices for paid meals in non-CEP schools rose, due to the Paid Lunch Equity requirement) and 
related operational issues with having both CEP and conventional schools. Concern about the impact 
of the CEP on funding for educational programs was shared by all three groups of LEAs surveyed, 
with the participating LEAs being most often concerned about this issue. According to one State, this 
issue undercut the potential for the CEP to reduce burden, because it created new paperwork for the 
LEA (as discussed below). One-quarter of eligible non-participating LEAs reported that they had not 
been informed about the CEP by their State; this lack of effective communication may have 
contributed to negative perceptions about the CEP. State and LEA data indicated that the transition to 
the CEP was easier for schools already serving free meals to all students under Provision 2 or 3, but 
these schools were relatively rare in the CEP States. 

3.4.3 Impacts of the CEP on Allocation of Educational Funds 

Data from both interviews and surveys indicate that States and LEAs expressed deep concern about 
the impacts of the CEP on Federal and State funding for educational programs (e.g., Title I, E-rate, 
and State aid to schools) to and within LEAs. Ten of 11 States perceived this as the largest barrier to 
adopting and implementing the CEP, and it also appeared to be the largest concern among 
participating LEAs. 

Federal Title I policy allows LEAs to use the claiming percentage for each school (based on the ISP 
for the school, group of schools, or entire LEA) in lieu of the FRP student percentage normally based 
on both identified students and household applications. This approach can result in funds shifting to 
schools newly eligible for Title I funds away from previous recipients. Ten of 11 States reported that 
a sizable number of LEAs had major concerns about the effect of the CEP on Title I allocations, and 
five of these States reported that this issue had affected LEA participation. 

Nine of the 11 States indicated that they had communicated this year with other divisions within the 
State Department of Education about the effect of CEP on other programs (e.g. Title I, E-rate, No 
Child Left Behind accountability 41 ) that use free and reduced price data. Although two States reported 
that they have resolved these data concerns, the others indicated that there were still unaddressed 
issues at the State level. 

In response to the funding issues, some LEAs limited the CEP to schools already receiving the 
maximum Title I funds, or to schools that did not depend on Title I funds (such as special education 


41 The No Child Left Behind Act requires reporting on adequate yearly progress for economically 

disadvantaged students. These students are usually identified on the basis of FRP meals certification. The 
Department of Education has instructed LEAs to identify all students in CEP schools as economically 
disadvantaged. 
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programs). As indicated by the analysis presented earlier in this chapter, however, the vast majority of 
LEAs that participated in CEP did so in all schools. 

Nine States reported that LEAs were using alternative household income form s to replace FRP meal 
applications. In seven of these States, the forms were optional for all CEP LEAs. One State mandated 
collection of household income forms for CEP LEAs, and one State effectively made the form s 
necessary for CEP LEAs because compensatory education funding depended on the data. Six of the 
nine States provided standardized forms for LEA use, and two of the other three were in the process 
of developing a standardized form. Two States chose not to offer the option of a household income 
form, because the State determined that existing Federal policy adequately addressed the issues raised 
by eliminating FRP meals applications, and participating LEAs were not collecting household income 
data for students in CEP schools. 

Telephone surveys of Title I directors and other State administrators responsible for the State Title I 
programs (in all 50 States and the District of Columbia) provided further information about how FRP 
data are used for multiple purposes. As shown in Exhibit 3.20, a substantial majority (90 percent) of 
States reported using FRP data to identify economically disadvantaged students for reporting 
requirements (e.g., adequate yearly progress) for No Child Left Behind (NCLB). In addition, almost 
three-quarters of respondents (71 percent) reported that they used FRP data for other education 
planning purposes such as school-level plans, preschool education, decisions regarding grants, and 
staff development. 


Exhibit 3.20: Percentages of States Reporting Use of FRP Data and Other Data Sources 


Use FRP Data: 

Percent of States 

To identify economically disadvantaged students for NCLB 

90.2% 

For planning purposes related to education 

70.6% 

To allocate Title 1 funds to LEAs 

58.8% 

To allocate Title 1 funds to schools 

33.3% 

To waive fees for low income students 

15.7% 


Sources: Based on telephone interviews conducted in spring and summer 2013 with Title I administrators in 51 
States. Multiple responses were allowed, so percentages may sum to more than 100. 


The majority of State Title I directors (59 percent) indicated that they used FRP data to allocate Title I 
funds to LEAs. About one-quarter of respondents voluntarily reported that they used U.S. Census 
poverty data in addition to or in combination with FRP data to make these allocation decisions and 
five respondents reported using a combination of factors. 42 One-third of States that they use FRP data 
to allocate Title I funds to schools, but the majority of States (65 percent) explicitly reported that they 
did not use FRP data this way. 43 Of the States who reported they did not use FRP data to allocate 


4_ According to the U.S. Department of Education, Title 1 funds are allocated to LEAs based on Census 
estimates of the number of children in poverty (U.S. Department of Education, 2011). The response that 
States use FRP data to allocate Title I funds to LEAs is therefore questionable. As noted in the text, about 
one-quarter of States also indicated that they use Census data for Title I allocation to LEAs. Further 
investigation into State procedures would be required to interpret this finding and is beyond the scope of 
this study. 

43 One respondent did not know whether his/her State used FRP data to allocate Title 1 funds to schools. 
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Title I funds to schools, more than half commented that LEAs allocate Title I funds to schools. In 
addition to the State Child Nutrition and Education offices, more than one-third of States (39 percent) 
reported that other State agencies, such as the Department of Social Services, Human Services, and 
Health and Family Services, also relied on the FRP data. 

Almost one-half of States (49 percent) reported that LEAs in their State used the FRP data for 
purposes not mandated by the State, most commonly for grant applications. Other reported uses of the 
FRP data by LEAs (as reported by States) included student fee waivers (e.g., books, sports) and other 
programs (e.g., tutoring). 

Approximately two-thirds of the Title I directors indicated that they would consider alternative data 
sources to replace FRP data. Most States (71 percent) were not aware of the American Community 
Survey (ACS), which collects household income information and might be used to estimate poverty 
rates for school districts or groups of schools, although important issues with this approach have been 
identified (National Research Council, 2012). The remainder of States were divided nearly equally 
between those who would and would not consider using the ACS (16 percent and 14 percent). 

Outside of the ACS, the three most appealing data sources reported by States were TANF and/or 
direct certification data (29 percent), U.S. Census poverty data (20 percent) and household survey 
data (10 percent). The largest drawback to the ACS and U.S. Census poverty data reported by States 
was that it did not provide individual student level data. 

3.4.4 Summary of Implementation Results 

In summary, three States and their LEAs succeeded in implementing the CEP in SY 2011-12, four 
more States and their LEAs did so in SY 2012-13, and four additional States were well-positioned to 
implement during SY 2013-14 despite numerous challenges. Some States and LEAs had already 
noticed increased meal participation, and according to States, LEAs were excited about the CEP. Key 
challenges for CEP States and LEAs were (1) the limited time during the initial period of 
implementation to understand the CEP, make decisions about participation, and implement it, and (2) 
understanding and trying to address the implications of the CEP for educational programs that use 
individual student FRP meals certification data. At the LEA level, the biggest barriers were financial 
concerns: uncertainty about the impacts of the CEP on NSLP and SBP participation, and the impacts 
on LEA finances. Equity issues and operational challenges (especially for LEAs with mixes of 
eligible and non-eligible schools) were also important concerns. Addressing the timing and 
information issues is a relatively straightforward problem, but unless States find ongoing alternatives 
for meeting the data needs of other Federal and State programs, this will likely be an ongoing 
challenge. 
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4. Take-Up of the Community Eligibility Provision 


This chapter describes the characteristics of LEAs and schools that took up the Community Eligibility 
Provision (CEP) in SY 2012-13. It also describes how these characteristics differ for other high- 
poverty LEAs and schools that did not take up the CEP. This description helps to fulfill the 
Congressional mandate for the Implementation Study (Research Objectives 1 and 3, as discussed in 
Chapter 2) and to provide descriptive information as context for the results of the Impact Study. The 
first section presents counts of near-eligible, eligible non-participating, and participating LEAs, 
overall and by State. (See Chapter 2 for definitions of these three groups as used in this report.) It also 
presents data on LEA approaches to claiming percentages. The second section provides descriptive 
statistics to address the question of how these three groups differ, both across and within the seven 
States. This section also describes near-eligible, eligible non-participating, and participating schools. 
The final section presents the results of a multivariate statistical analysis intended to isolate the 
distinct contribution of factors associated with LEA take-up of the CEP. Analyses presented in this 
chapter used administrative data on LEAs and schools; the text identifies the relevant populations for 
the descriptive and multivariate analyses. 

4.1 Numbers of Near-Eligible, Eligible Non-Participating, and Participating 
LEAs 

The analysis in this section and the next distinguishes between two categories of LEAs within the 
universe of those that participate in the NSLP (with or without the SBP). The first category is labeled 
as “regular and charter LEAs” and consists of typical public school districts, referred to as “regular” 
LEAs by the Department of Education’s Common Core of Data (CCD), and charter school 
organizations treated as distinct LEAs, as was the case in Michigan, Ohio, New York, and the District 
of Columbia. Illinois incoiporates charter schools within regular LEAs, and Kentucky and West 
Virginia have no charter schools. The second category, called “other LEAs,” includes non-public 
schools or groups of non-public schools (such as Catholic dioceses), and non-typical public LEAs, 
including regional education service agencies, State-operated agencies, and other types. This 
distinction is relevant for two reasons. First, the regular and charter LEAs, but not the other LEAs, are 
included in the analysis of the impacts of the CEP for reasons discussed in Chapter 2. Second, data 
are missing for substantial numbers of the other LEA category, as described in Appendix 4A. 

4.1.1 Regular LEAs: Numbers of Near-Eligible, Eligible Non-Participating, and Participating 

Exhibit 4.1 shows the counts of near-eligible, eligible non-participating, and participating LEAs by 
State for the regular and charter LEAs, based on data for SY 2012-13. In the seven States where the 
CEP was implemented, there were 1,373 regular and charter LEAs that were at least near-eligible, 
representing 39 percent of the total of 3,533 public LEAs in the SY 2011-12 VSR data for these 
seven States. 

Among the three groups of regular and charter LEAs represented in Exhibit 4.1, more than half (743) 
were eligible but non-participating, about a fourth (349) were participating, and the remainder (281) 
were near-eligible. Eligible non-participating LEAs constituted the largest of the three groups in 
Illinois, Kentucky, Michigan, and Ohio. Participating LEAs made up the largest group in the District 
of Columbia and West Virginia, while near-eligible LEAs represented the largest group (more than 
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one-half of the total) in New York. 44 Near-eligible LEAs comprised about a third of the total for the 
three groups in Illinois and Ohio but only a small fraction in the other four States. 

The participation rates among eligible regular and charter LEAs, as shown in Exhibit 4.1, provide a 
measure of the take-up of the CEP. By this measure, about one-third of eligible LEAs in the seven 
States took up the CEP for SY 2012-13. Participation rates varied across States, from 24 percent in 
Michigan to over 60 percent (61 and 65 percent, respectively) in the District of Columbia and West 
Virginia. The lowest participation rates were in the three largest States (in order, Michigan, Ohio, and 
Illinois, as measured by the LEA totals in Exhibit 4.1). Possible explanations for differences in 
participation rates are discussed later in this chapter. 


Exhibit 4.1: Regular and Charter LEAs, by CEP Status and State (SY 2012-13) 



a Eligible LEAs include eligible non-participating and participating. 

b Totals includes only LEAs that were near-eligible, eligible non-participating, or participating in the CEP. 

Sources: FNS Verification Summary Report (FNS-742) Database, State lists of participating LEAs. Classification 
of regular and charter LEAs based on Common Core of Data. 

4.1.2 Other LEAs: Numbers of Near-Eligible, Eligible Non-Participating, and Participating 

Exhibit 4.2 shows the counts of near-eligible, eligible non-participating, and participating LEAs by 
State for the “other LEAs.” This category includes non-public LEAs, regional education service 
agencies, and other non-typical public education agencies. A total of 318 other LEAs were near- 
eligible, eligible non-participating, or participating in the CEP. Non-public LEAs comprised about 
two-thirds of the total number of other LEAs (209 of 318). 45 New York contributed the largest 
number of participating LEAs (51) in this category, followed by Illinois (7). Review of the detailed 


44 Only 6 percent of all LEAs in New York have ISPs above 40 percent, according to the VSR data for SY 
2011-12, much lower than other States. As a result, a large State like New York has so few LEAs falling 
into these three groups in Exhibit 4.1. However, direct certification in New York was reported to be highly 
effective, with estimates for SY 2011-12 indicating that 100 percent of school-age SNAP children were 
directly certified (Moore et al., 2012). The discrepancy between these statistics might be explained by high 
concentrations of SNAP children in a small number of large LEAs, large numbers of heterogeneous LEAs 
(so that identified students are dispersed across LEAs), or a combination of these characteristics. Also, the 
number of eligible LEAs in New York may be underestimated because school-level ISP data for non¬ 
participating LEAs were not available. 

45 For comparison, the SY 2011-12 VSR data for these seven States included 1,131 non-public LEAs. This 
count does not include public LEAs that are not regular or charter LEAs. Thus near-eligible LEAs 
represented approximately 18 percent of non-public LEAs in the seven States (209 out of 1,131). 
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LEAs lists (not included in this report) indicates that the two other LEAs with the most schools were 
the Archdiocese of Chicago (198 schools) and the Catholic Diocese of Cleveland (66 schools). Both 
were eligible but non-participating in the CEP. 

On average, over two-fifths (44 percent) of eligible other LEAs participated in the CEP, a higher rate 
than did regular and charter LEAs. Across the States, participation rates for eligible other LEAs 
varied widely, from 15 percent or less in Illinois and Ohio to 80 percent or more in the District of 
Columbia and New York. 

The State-level data provide perspective on the distinct types of LEAs that make up the population of 
other LEAs described in Exhibit 4.2. Illinois, Michigan, and Ohio had roughly an even balance of 
public and non-public LEAs within the broad category of other LEAs. In New York, nearly all other 
LEAs (96 of 99) were non-public, many of which were religious schools. Kentucky, West Virginia, 
and the District of Columbia had relatively few LEAs in this group, and they were all non-public. 

Exhibit 4.2: Other LEAs, by CEP Status and State (SY 2012-13) 



Other LEAs include non-public, regional education service agencies, and other public education agencies as 
identified by the Common Core of Data. 

a Totals include only LEAs that were near-eligible, eligible non-participating, or participating in the CEP. 

Sources: FNS Verification Summary Report (FNS-742) Database, State lists of participating LEAs. 

4.1.3 Eligible School Participation in CEP LEAs 

Exhibit 4.3 presents the proportion of eligible schools in participating LEAs that actually offered the 
CEP. This table distinguishes between those LEAs that were eligible to participate LEA-wide—those 
for which the ISP for the LEA was at least 40 percent—and those LEAs within which only some 
schools were eligible to participate. Either type of LEA could choose to participate at a subset of 
eligible schools, provided that the school or group of schools had an average ISP of at least 40 
percent. As discussed in Chapter 3, some participating LEAs chose to implement the CEP in some but 
not all eligible schools for a variety of reasons. 

Among participating LEAs that were eligible to participate LEA-wide, a very high proportion of 
schools offered the CEP: among these LEAs, 97 percent of schools offered the CEP, on average, and 
92 percent of LEAs offered the CEP at all schools. The average proportion of schools that offered the 
CEP ranged from 92 percent in Kentucky to 100 percent in the District of Columbia and West 
Virginia for participating LEAs that were eligible LEA-wide. 

Participating LEAs that were not eligible LEA-wide offered the CEP in a smaller proportion of 
eligible schools than the LEAs that were eligible LEA-wide. Among participating LEAs that were not 
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eligible LEA-wide, on average 79 percent of eligible schools offered the CEP, and 53 percent offered 
the CEP at all eligible schools. The proportion of eligible schools participating ranged from 77 
percent in West Virginia to 100 percent in the District of Columbia. 


Exhibit 4.3: Percent of Eligible Schools Participating in CEP in Participating LEAs, by LEA- 
Wide Eligibility and State (SY 2012-13) 



West Virginia_100_77_85 


a New York data are missing number of eligible schools for 2 small LEAs that are not eligible LEA-wide. 

b Ohio data are missing number of eligible schools for all of the LEAs that are not eligible LEA-wide (affects 5 
LEAs). 

Sources: FNS Verification Summary Report (FNS-742), State lists of participating LEAs. 

4.1.4 LEA Approaches to Claiming Percentages 

Schools can establish claiming percentages for the CEP individually, as part of a group, or on an 
LEA-wide basis, depending on the ISPs of the LEAs’ schools. How schools qualify for the CEP has 
implications for how the LEAs operate the CEP. If the LEAs establish CEP claiming percentages on 
an LEA-wide basis, then LEAs can operate all schools under the same rules, including those 
regarding FRP meals applications and claims for reimbursement. On the other hand, establishing 
claiming percentages on the basis of individual schools or groups of schools means that the LEAs 
must have different procedures for CEP schools and non-CEP schools as well as among CEP schools. 

Exhibit 4.4, below, presents the distribution of regular LEAs by their mode of establishing claiming 
percentages for the CEP, with counts in Panel A and percentages of LEAs in each State in Panel B. 46 
For example, the majority of participating LEAs established their claiming percentage on an LEA- 
wide basis, as shown in Exhibit 4.4, with the proportion of LEAs doing so ranging from 34 percent in 
West Virginia to 79 percent in Ohio. Roughly a third of participating LEAs in Michigan and West 
Virginia established claiming percentages for the CEP based on groups of schools; in other States, the 
percentage of participating LEAs that established a claiming percentage on this basis was about 20 


46 The categories of claiming percentage map closely onto whether an LEA is participating in the CEP with 
some or all of its schools. LEAs that establish a claiming percentage on an LEA-wide basis are all 
participating in the CEP hilly, with all schools participating. The majority of LEAs that establish their 
claiming percentage on a group basis do not have the census of schools participating in the CEP, with three 
exceptions. All LEAs that establish their claiming percentage on an individual school or mixed basis are 
not participating in the CEP with all schools. Slightly less than half (45 percent) of LEAs that establish 
their claiming percentage on an individual school basis have more than one participating school. 
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percent or less. In Kentucky and West Virginia, about 20 percent of participating LEAs established 
claiming percentage on an individual school basis; in other States, this percentage was 13 percent or 
less. A few (4) participating LEAs established claiming percentages through a combination of 
methods, generally grouped and individual schools. 


Exhibit 4.4: Distribution of Participating LEAs by Mode of Establishing Claiming Percentage 


Mode of Establishing 
Claiming Percentage 

DC 

IL 

KY 

Ml 

NY 

OH 

wv 

Total 

Panel A: Number of participating LEAs 

Entire LEA 


29 

28 

57 

10 

54 

12 

190 

Group 


10 

9 

34 

3 

6 

12 

74 

Individual 


6 

10 

10 

1 

2 

8 

37 

Combination 



1 




3 

4 

Missing 

19 

3 

3 

6 

3 

6 


40 

Total 

19 

48 

51 

107 

17 

68 

35 

345 

Panel B: Percent of participating LEAs 

Entire LEA 


60.4 

54.9 

53.3 

58.8 

79.4 

34.3 

55.1 

Group 


20.8 

17.6 

31.8 

17.6 

8.8 

34.3 

21.4 

Individual 


12.5 

19.6 

9.3 

5.9 

2.9 

22.9 

10.7 

Combination 


0.0 

2.0 

0.0 

0.0 

0.0 

8.6 

1.2 

Missing 

100.0 

6.3 

5.9 

5.6 

17.6 

8.8 

0.0 

11.6 

Total 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 


Source: State lists of participating LEAs and schools. 


4.2 Characteristics of Near-Eligible, Eligible Non-Participating, and 
Participating LEAs and Schools 

This section presents selected characteristics of near-eligible, eligible non-participating, and 
participating LEAs and schools, focusing on the regular and charter LEAs for all seven States. 
(Characteristics of LEAs other than regular and charter LEAs are not presented because of the data 
limitations, as discussed in Section 4.1.) The section begins with overall patterns at the LEA level, 
then considers variations in the LEA-level patterns among States, and finally presents the patterns at 
the school level. 

4.2.1 Overall Patterns of the Characteristics of Near-Eligible, Eligible Non-Participating, and 
Participating LEAs 

Among regular and charter LEAs in the seven CEP States, there were large differences between 
participating and eligible, non-participating LEAs on several characteristics, as shown in Exhibit 4.5. 
Compared with eligible non-participating LEAs, participating LEAs, on average, had more students 
(7,210 versus 2,712), more schools (15 versus 6), and higher ISPs (50 versus 38 percent). 47 The 
higher ISPs of participating LEAs suggest that decisions to participate in the CEP reflect incentives in 
the reimbursement formula. As the ISP approaches 62.5 percent, the proportion of meals reimbursed 
at the free rate approaches 100 percent, and thus it becomes more financially advantageous to 
implement the CEP. 


47 As described in section 2.1, an LEA can be considered eligible if it has at least one school with an ISP of 
40 percent or more. Thus, eligible LEAs can have ISPs less than 40 percent. 
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Participating LEAs, on average, were more often urban (35 versus 23 percent), had more Black 
students (36 versus 24 percent), and were more likely to include charter schools (32 versus 25 
percent). On the other hand, there was no notable difference between the two types of LEAs in their 
fractions of Title I schools or the distribution of students among grade levels. 48 Participating LEAs 
had only modestly higher percentages of students eligible for FRP meals (70 versus 66 percent, on 
average). 

Although participating LEAs had substantially more students than eligible non-participating LEAs, 
on average, this difference was essentially driven by a few participating LEAs with large enrollments. 
(Six of the ten largest LEAs in the seven States—including New York City, Chicago, Detroit, and 
Cleveland—are participating.) The difference between participating and eligible non-participating 
LEAs in the percentage with enrollments of 5,000 or more was less dramatic (15 versus 12 percent), 
and there was almost no difference in the percentage with enrollments of 2,500 or more (31 versus 29 
percent). 

Differences between eligible non-participating and near-eligible LEAs were similar but less dramatic 
than those between participating and eligible non-participating LEAs. Most notably, eligible non¬ 
participating LEAs had more students (2,712 versus 1,438) and schools (6 versus 3). Relative to near- 
eligible LEAs, eligible non-participating LEAs also had slightly higher ISPs, FRP percentages, and 
percentages of Black students, and they were slightly more likely to include charter schools and to be 
located in a city. Differences in ISPs between near-eligible and eligible LEAs are, of course, the 
direct result of the definition of eligibility for the CEP. Unlike the average enrollment disparity 
between participating and eligible non-participating LEAs, the enrollment difference between eligible 
non-participating and near-eligible LEAs was not simply an artifact of a few very large LEAs 
skewing the result. Compared to near-eligible LEAs, eligible non-participating LEAs had much 
higher percentages of LEAs with enrollments of 5,000 or more (12 versus 5 percent) and 2,500 or 
more (29 versus 17 percent). 


48 Exhibit 4.9 shows the distribution of schools by grade span. 


pg. 70 | 4. Take-Up of the Community Eligibility Provision 


Abt Associates 





Community Eligibility Provision Evaluation: Final Report 


Exhibit 4.5: Regular and Charter LEA Characteristics—All States 



Near-Eligible 

Eligible, Non- 
Participating 3 

Participating 15 

Combined 6 


Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Number of LEAs with VSR data c 

276 


737 


345 


1,358 


Number of LEAs with CCD data 

281 


743 


349 


1,373 


Enrollment 

1,438 

(1,651) 

2,712 

(5,684) 

7,210 

(59,308) 

3,593 

(30,208) 

Enrollment 1-499 (%) 

37 


27 


33 


31 


Enrollment 500-2499 (%) 

47 


45 


36 


43 


Enrollment 2500-4999 (%) 

12 


17 


16 


15 


Enrollment 5000+ (%) 

5 


12 


15 


11 


Percentage distribution of students 
in grades K-12 

Percent in grades K-5 

51 

(21) 

49 

(18) 

52 

(21) 

50 

(19) 

Percent in grades 6-8 

25 

(13) 

24 

(10) 

23 

(12) 

24 

(11) 

Percent in grades 9-12 

24 

(24) 

27 

(20) 

25 

(21) 

26 

(21) 

Number of schools' 5 

3 

(3) 

6 

(9) 

15 

(104) 

7 

(53) 

1 school (%) 

40 


30 


36 


33 


2-5 schools (%) 

46 


39 


28 


38 


6-14 schools (%) 

13 


24 


24 


22 


15+ schools (%) 

1 


6 


12 


7 


Identified Student Percentage (ISP) b 

35 

(10) 

38 

(15) 

50 

(19) 

40 

(16) 

Students free/reduced lunch (%) e 

65 

(15) 

66 

(19) 

70 

(19) 

67 

(18) 

Any charter schools (%) 

21 


25 


32 


26 


Percent Title 1 schools 

92 

(19) 

86 

(20) 

85 

(20) 

87 

(20) 

Urban LEA (%) 

21 


23 


35 


26 


Percent students Black 

21 

(31) 

24 

(34) 

36 

(38) 

27 

(35) 

Percent students Hispanic/Latino 

8 

(14) 

7 

(12) 

7 

(11) 

7 

(12) 


a Eligible non-participating LEAs include those with some but not all schools with official ISP of 40% or more. 

b Combined means and percentages include only LEAs and schools that were near-eligible, eligible non¬ 
participating, or participating in CEP. 

c VSR data for a few LEAs were not used because they were substantially inconsistent with State or CCD data. 
d The large SD for the number of schools and enrollment is due to the inclusion of several very large LEAs, 
including New York City. 

e To provide context for the free/reduced lunch percentages, data from the 2009-10 CCD indicates that the 
average public school district had 46 percent of students eligible for free/reduced lunch meals. 

Sources: FNS Verification Summary (FNS-742) data, State lists of participating LEAs, Common Core of Data. 

Exhibit 4.6 shows the distribution of projected ISPs (based on October 2011 VSR data) for LEAs by 
their CEP eligibility and participation in the 2012-13 school year. As expected, most near-eligible 
LEAs had ISPs below 40 percent. Some near-eligible LEAs had ISPs above 40 percent because of the 
way that eligible LEAs were defined. 49 Most eligible non-participating LEAs had ISPs between 20 
and 50 percent. Less than 20 percent of eligible non-participating LEAs had ISPs above 50 percent. 


49 As discussed in Chapter 2, eligible non-participating LEAs in New York and Ohio were identified on the 
basis of the lists compiled by the respective State. LEAs with ISPs above 40 percent that were not on these 
lists were not notified of eligibility and thus were not considered to have made a choice regarding 
participation in the CEP. In other States, eligible non-participating LEAs were identified based on State- 
provided school-level ISP data. In some cases, incomplete school ISP data resulted in LEAs with a ISP 
above 40 percent being identified as near-eligible rather than eligible. 
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Participating LEAs generally had ISPs above 40 percent. Not all participating LEAs had ISPs above 
40 percent, because participation requires only one eligible school in the LEA. These data suggest 
that having a district ISP above 50 percent was strongly associated with an LEA’s likelihood of 
participation in the CEP. 

Exhibit 4.6: Distribution of Projected ISPs for LEAs by CEP Eligibility and Participation—All 
States_ 

Percentage of LEAs 


Eligible, Non- 


ISP 

Near-Eligible 

Participating 

Participating 

Combined 3 

<20 

5.4 

7.9 

4.4 

6.5 

20-39 

7.6 

20.5 

5.2 

14.0 

30-39 

71.7 

31.1 

21.2 

36.9 

40-49 

9.4 

23.9 

22.1 

20.5 

50-59 

3.3 

9.1 

18.9 

10.4 

60-69 

1.5 

4.5 

15.1 

6.6 

>70 

1.1 

3.1 

13.1 

5.2 

Total 

100.0 

100.0 

100.0 

100.0 


a Combined means and percentages include only LEAs and schools that were near-eligible, eligible non¬ 
participating, or participating in CEP. 

Sources: FNS Verification Summary (FNS-742) data, State lists of participating LEAs. Projected ISP based on 
October 2011 VSR data. 

Eligible LEAs and schools were more likely to participate in the CEP if they had high percentages of 
students certified for FRP meals. As shown in Exhibit 4.7, close to half of eligible LEAs with over 70 
percent FRP students participated in the CEP, while less than a quarter of those with 60 percent FRP 
students or less did so. The differences in participation rates by FRP student percentage were larger at 
the school level: among all schools in eligible LEAs, less than one in six schools participated where 
the school FRP percentage was 60 percent or less, while over half of schools with more than 80 
percent FRP students participated. 


Exhibit 4.7: CEP Participation Rate for Eligible Regular LEAs and Schools, by Percentage of 
Students Certified for Free or Reduced-Price (FRP) Meals 


Pet. FRP Students 

LEAs 

Schools 

<51 

14.0 

5.8 

51-60 

22.5 

15.4 

61-70 

27.5 

27.4 

71-80 

47.0 

37.6 

>80 

44.0 

51.4 

Total 

31.9 

28.6 


a For this table schools are considered eligible if they belong to an eligible LEA. 

Sources: FNS Verification Summary (FNS-742) data for LEA percent FRP (October 2010 for Year 1 States, 
October 2011 for Year 2 States), Common Core of Data (SY 2010-11) for school percent FRP, State lists of 
participating LEAs and schools. Excludes LEAs other than regular school districts and charter schools. 

The relationship of the FRP student percentage to the ISP is an important consideration in assessing 
the appropriateness of the multiplier that is used to determine the percentage of meals claimed at the 
free rate in CEP schools (the claiming percentage). The HHFKA set the multiplier at 1.6 but gave 
USDA the authority to set it between 1.3 and 1.6. This multiplier is also used to estimate the FRP 
student percentage for programs that use this statistic when identifying high-poverty schools and 
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allocating funds among schools (e.g., Title I). In this context, the programs assume that if, for 
example, a school has an ISP of 40 percent, the FRP student percentage would be 64 percent if the 
school took applications for FRP meals. 

There are two reasons to consider using a lower multiplier. First, at the current multiplier, 
participating LEAs might get more Federal revenue than they would under conventional claiming 
procedures. It may be desirable to allow CEP LEAs to get some additional Federal funds to offset the 
lack of revenue from student payments. This incentive would encourage LEAs to take up the CEP 
and may result in higher student participation in NSLP and SBP. However, too much shift of funding 
from local to Federal sources might be a concern. Second, if the current multiplier did not accurately 
represent the relationship of the FRP student percentage to the ISP, the use of the claiming percentage 
as the proxy for the expected FRP student percentage would distort statistics and funding allocation 
for programs that use this measure. 

In the typical participating LEA, the FRP student percentage was 1.6 times the ISP (as indicated by 
the median shown in Exhibit 4.8), while for all eligible LEAs this ratio was 1.8. (See the notes to the 
exhibit for the reference period.) Thus, the multiplier appears to be an accurate reflection of the 
relationship of the FRP student percentage to the ISP in the typical participating LEA, but it appears 
to understate the total number of FRP students (relative to the subset of identified students) in the 
typical non-participating LEA. This finding is consistent with a sorting process where LEAs choose 
to implement the CEP only if meal reimbursements under the CEP will be comparable to or greater 
than reimbursements under conventional procedures. However, each LEA would make this decision 
based on its own ISP, FRP student percentage, participation rates among FRP versus paid students, 
and available resources to cover lost revenue. 

The ratio of the FRP student percentage to the ISP (the FRP/ISP ratio) varied substantially among 
both eligible non-participating and participating LEAs, as shown in Exhibit 4.9. At the extremes, 
nearly 3 percent of eligible non-participating LEAs and over 10 percent of participating LEAs had 
ratios less than 1.3, while over 13 percent of eligible non-participating LEAs and 9 percent of 
participating LEAs had ratios greater than 2.5. The mean ratio was 2.9 for eligible non-participating 
LEAs and 2.0 for participating LEAs. The substantial differences between the means and the medians 
indicate that the distribution of the FRP/ISP ratio is skewed, with wider range of values among the 
top 50 percent of LEAs than among the bottom 50 percent. Consistent with the difference in the 
median ratios, participating LEAs were more likely to have ratios less than 1.6 and less likely to have 
ratios above 1.6 than eligible non-participating LEAs. 

The statistics in Exhibit 4.8 and 4.9 should be viewed with caution for four reasons. First, they are 
based on Verification Summary Report (FNS-742) data, which may not provide the best measure of 
the ISP and its relationship to the FRP student percentage. 50 Second, these statistics represent the 
seven States selected by FNS as early implemented of the CEP and likely are not representative of all 
LEAs nationwide. Third, the analysis of FRP/ISP ratios does not fully inform considerations about 
the suitability of the multiplier for determining the claiming percentage under the CEP. To do this 


0 As discussed in Chapter 3, the FNS-742 data provide the ISP as of October. LEAs can increase their ISPs 
from the level indicated in the FNS-742 data by conducting additional direct certification later in the school 
year, and in some cases this process identifies students previously approved by application. Reporting error 
in the FNS-742 data may also contribute to extremely high or low values of the FRP/ISP ratio. 
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assessment requires taking into account the differential participation rates of students approved for 
free, reduced-price, and paid meals. Chapter 5 addresses this issue by providing estimates of the 
impact of the CEP on the Federal reimbursement per meal for NSLP and SBP. Fourth, the choice of 
the multiplier is ultimately a policy decision, balancing the goal of expanding NSFP and SBP 
participation with potential concerns about equitable levels of reimbursement for CEP and non-CEP 
LEAs. Such a decision would need to take into account the distribution of the potential impacts of 
changing the multiplier across all eligible LEAs nationwide, not just the impact on the typical 
(median) LEA. 


Exhibit 4.8: Median and Mean Ratio of FRP Student Percentage to ISP for Eligible Non- 
Participating, Participating, and All Eligible Regular and Charter LEAs 




Percent distribution 


%FRP/ 




ISP 

Eligible, Non-Participating 

Participating 

All Eligible LEAs 

Median 

1.9 

1.6 

1.8 

Mean 

2.9 

2.0 

2.6 


Sources: FNS Verification Summary (FNS-742) data (October 2010 for Year 1 States, October 2011 for Year 2 
States), State lists of eligible and participating LEAs. Excludes LEAs other than regular school districts and 
charter schools. 


Exhibit 4.9: Distribution of Eligible Non-Participating, Participating, and Ail Eligible Regular 
and Charter LEAs by Ratio of FRP Student Percentage to ISP 


Sources: FNS Verification Summary (FNS-742) data (October 2010 for Year 1 States, October 2011 for Year 2 
States), State lists of eligible and participating LEAs. Excludes LEAs other than regular school districts and 
charter schools. 
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4.2.2 State Variations in the Characteristics of Near-Eligible, Eligible Non-Participating, and 
Participating LEAs 

Most of the differences among the three groups of LEAs (refer to Exhibit 4.5) hold at least roughly 
for individual States. Exhibit 4.10 displays in chart form the State-level descriptive statistics that 
differ the most for the three LEA groups. (Appendix 4A: State Tables of LEA and School 
Characteristics presents the supporting data for each State individually.) 51 Among the available 
characteristics, the most consistent differences among the three groups of LEAs were in ISPs. In all 
seven States, participating LEAs had higher ISPs than eligible non-participating LEAs. These gaps 
were substantively large (about 8 percentage points or more) in all States except Ohio and West 
Virginia. Eligible non-participating LEAs, in turn, had higher ISPs than near-eligible LEAs in six of 
the seven States, as expected given the program rules. 52 The gaps in ISPs between eligible non¬ 
participating and near-eligible LEAs were largest in Illinois, Ohio, and West Virginia. In all States 
except Illinois, participating LEAs had higher percentages of students certified for FRP meals than 
eligible non-participating LEAs. 


51 Appendix 4A also provides tabulations of the characteristics of “other” LEAs (those that are not regular or 
charter public LEAs) in all seven States (combined) and the distribution of LEA ISPs for each State. See 
Exhibit 4A.10 for the median FRP/ISP ratio by State. 

2 The exception is New York; see footnote to discussion to Exhibit 4.5. 
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Exhibit 4.10: Characteristics of Near-Eligible, Eligible Non-Participating, and Participating 
LEAs by State 

Identified student percentage 
(average) 

100 - 


75 



ALL DC IL KY Ml NY OH WV 


Enrollment (average, in 1000s) 




■ Near-Eligible ■ Eligible Non-Participating 


Participating 
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Exhibit 4.10 (continued) 


Percent free/reduced lunch 
(average) 



ALL DC IL KY Ml NY OH WV 


Percent of LEAs that are in an urban area 
(average) 



ALL DC IL KY Ml NY OH WV 


Percent of students who are black 



ALL DC IL KY Ml NY OH WV 


■ Near-Eligible ■ Eligible Non-Participating Participating 

Notes: 

1. The average enrollment shown for participating LEAs in New York was truncated. The actual average 
enrollment for the group of LEAs was 68,529. 

2. Kentucky and West Virginia had no charter schools. Illinois had charter schools, but they were included in 
regular LEAs. 

Source: FNS Verification Summary (FNS-742) data, State lists of participating LEAs, Common Core of Data. 
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Considering other characteristics with notable differences across groups, participating LEAs had 
more students enrolled, on average, than eligible non-participating LEAs in all of the States except 
Kentucky. There was a particularly shaip contrast in size between participating and eligible non¬ 
participating LEAs in the District of Columbia, Illinois, and New York (all States where the largest 
public school districts participated). Participating LEAs were more often urban than eligible non¬ 
participating LEAs overall, but this pattern held in only three States (Ohio, New York, and 
Michigan). (The District of Columbia is all urban, so this statistic showed no variation across groups.) 
Compared with the other groups of LEAs, participating LEAs had higher proportions—generally 
much higher—of Black students in all States except Ohio. 

Overall, participating LEAs more often had charter schools than eligible non-participating LEAs, 
which, in turn, more often had charter schools than near-eligible LEAs. 53 Michigan and Ohio 
contributed most of the LEAs with charter schools across the seven States. Illinois and Ohio conform 
to the overall pattern of LEAs with charter schools being associated with CEP eligibility and 
participation, while other States’ patterns were mixed. The District of Columbia consists almost 
entirely of LEAs with charters (with the exception of the one regular public school district), so it had 
little variation among CEP eligibility and participation groups. In Michigan, participating LEAs had 
charter schools more often than eligible non-participating LEAs, but eligible non-participating LEAs 
had charter schools less often than near-eligible LEAs. Of all seven States, New York had the fewest 
LEAs with charter schools among LEAs participating in the CEP. 

It should be noted that there were large differences in the structural characteristics of LEAs among 
States. (The discussion that follows is based on the combined characteristics of participating, eligible 
non-participating, and near-eligible LEAs, as shown in the State-level tables in Appendix 4A, and on 
additional information gleaned from State LEA lists). In Kentucky and West Virginia, LEAs aligned 
with counties, and there were no charter LEAs. Other States had large numbers of very small LEAs 
consisting of a single school or less than 500 enrolled students, but Kentucky and West Virginia had 
almost none. 54 Michigan, Ohio, New York, and the District of Columbia all had considerable 
numbers of charter LEAs, which comprised a large proportion of their LEAs with very small 
enrollments. New York had large fractions of both small (less than 2,500 students) and large (more 
than 5,000 students) LEAs, with few in between. Illinois had many small LEAs—77 percent with 
fewer than 2,500 students and 23 percent consisting of a single school—but it had no all-charter 
LEAs, although some of its LEAs included charter schools. The District of Columbia was unique 
among the seven CEP States insofar as it comprised one large regular LEA with 122 schools and 34 
charter school LEAs, most consisting of a single school. 

Among all CEP LEAs participating in SY 2012-13, 291 were participating for the first time, and 129 
also participated in the prior year, as shown in Exhibit 4.11. The Year 1 States (Illinois, Kentucky, 
and Michigan) added a total of 92 in SY 2012-13. Only three LEAs (all in Michigan) dropped out of 
the CEP after SY 2011-12. 


53 As defined for this report, LEAs with charter schools included regular public school districts that include 
charter schools, independent charter schools, and groups of charter schools operated together as an LEA. 

4 LEAs in Kentucky and West Virginia are county-based, and therefore large. 
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Exhibit 4.11: Numbers of LEAs Participating in SY 2011-12, SY 2012-13, and Both Years 


2011-12 

only 

2011-12 and 
2012-13 

2012-13 only 

District of Columbia 

0 

0 

23 

Illinois 

0 

35 

20 

Kentucky 

0 

18 

34 

Michigan 

3 

76 

38 

New York 

0 

0 

68 

Ohio 

0 

0 

73 

West Virginia 

0 

0 

35 

Total 

3 

129 

291 


Source: State lists of participating LEAs. Includes regular, charter, and other LEAs. 


4.2.3 Characteristics of Near-Eligible, Eligible Non-Participating, and Participating Schools 

To complement the LEA-level description in the preceding sections, this section first presents the 
numbers of near-eligible, eligible non-participating, and participating schools in the seven CEP States 
and then considers the similarities and differences of these three types of schools. (See Chapter 2 for 
definitions of these types of schools.) 

The distribution of schools among the three types largely mirrored the distribution of LEAs. Three- 
fifths of all schools in the three groups (5,256) were eligible non-participating, about one-quarter 
(2,169) participate, and the rest (1,247) were near-eligible (Exhibit 4.12). Michigan, which had the 
most participating and eligible non-participating LEAs, also had the most participating and eligible 
non-participating schools. Counting the totals across all three groups, Ohio had more LEAs than 
Illinois, but fewer schools. This occurred because Ohio had a particularly large number of single¬ 
school LEAs (59 percent of LEAs across the three groups, as compared with 33 percent across the 
seven States). The counts for New York in Exhibit 4.12 exclude New York City. 55 

On average, across the seven CEP States, slightly less than one-third of the 7,425 eligible schools in 
regular and charter LEAs participated in the CEP, with rates ranging from 19 percent in Michigan to 
77 percent in the District of Columbia (where the city’s single regular public school district, a 
participating LEA, accounts for the great majority of all schools). As at the LEA level, school 
participation rates were lower in the States with the most schools in eligible or near-eligible LEAs. 


55 See Appendix 4A for discussion. 


Abt Associates 


4. Take-Up of the Community Eligibility Provision | pg. 79 















Community Eligibility Provision Evaluation: Final Report 


Exhibit 4.12: Regular and Charter Schools, by CEP Status and State (SY 2012-13) 



a Eligible LEAs and schools include eligible non-participating and participating. 

b Totals include only LEAs and schools that were near-eligible, eligible non-participating, or participating in the 
CEP. 

c New York City, which was partially participating with 301 CEP sites, is excluded from this analysis due to data 
problems. See Appendix 4A for details. 

Sources: FNS Verification Summary Report (FNS-742) Database, State lists of participating LEAs. Classification 
of regular and charter LEAs based on Common Core of Data. 

The patterns found for LEAs were generally replicated at the school level, although school-level 
differences between participating and eligible non-participating schools were sharper, most likely 
because only subsets of schools in some participating LEAs implemented the CEP, and these 
participating subsets of schools may have certain distinguishing characteristics. 

Compared to near-eligible and eligible non-participating schools, participating schools had much 
higher ISPs, FRP percentages, and percentages of Black students, as shown in Exhibit 4.13. 
Participating schools were also far more likely to have “other” or non-standard grade spans (that is, 
not K-5, 6-8, or 9-12). Most of these “other” grade spans were either combined elementary-middle 
or middle-high schools. Participating schools were more likely to be Title I schools or charter schools 
than eligible non-participating or near-eligible schools, although the difference in the charter school 
percentage was quite small. 
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Exhibit 4.13: School Characteristics in Regular and Charter LEAs—All States 



Near-Eligible 

Eligible, Non- 
Participating 

Participating 

Combined 


Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Enrollment 

473 

(336) 

491 

(372) 

415 

(243) 

469 

(340) 

ISP 11 

29 

(9) 

40 

(16) 

60 

(16) 

45 

(19) 

Students free/reduced 
lunch (%) 

55 

(21) 

59 

(23) 

79 

(18) 

64 

(24) 

Grade span 

K-5 (%) 

49 


49 


46 


48 


6-8 (%) 

21 


16 


10 


16 


9-12(%) 

19 


19 


12 


17 


Other (%) 

11 


15 


32 


19 


Title 1 schools (%) 

80 


76 


91 


81 


Charter schools (%) 

5 


5 


7 


6 


Percent students Black 

15 

(24) 

21 

(28) 

46 

(39) 

26 

(33) 

Percent students 
Hispanic/Latino 

7 

(13) 

12 

(21) 

9 

(18) 

10 

(19) 

Total number of schools' 1 

1,222 


4,793 


2,060 


8,075 



a Combined means include only LEAs that were near-eligible, eligible non-participating, or participating in CEP. 
b ISP not available for any near-eligible or eligible non-participating schools in New York, or for any near-eligible 
schools in Ohio. 


c Numbers of schools with enrollment data. Fewer schools had ISP and FRP percentage. Number of schools in 
CCD is less for near-eligible and participating, but more for eligible non-participating and combined. 

Sources: State lists of schools with ISPs and FRP percentages, Common Core of Data. 

The average ISP for eligible non-participating schools was 40 percent, and nearly half of the eligible, 
non-participating schools for which there were ISP data had ISPs less than 40 percent (based on 
analysis not shown in this report). This stems from the fact that all schools in eligible non¬ 
participating LEAs were classified as eligible non-participating schools, because these schools could 
be combined with other schools with higher ISPs in the same LEA to form groups of eligible schools. 
(About 8 percent of participating schools also had ISPs less than 40 percent.) Among schools with an 
ISP of 40 percent or higher, the average eligible non-participating school ISP was 52 percent, and the 
average participating school ISP was 63 percent. 

Across five of the seven States, school-level ISP was the characteristic most consistently and strongly 
associated with CEP eligibility and participation, as shown in Exhibit 4.14. (Appendix 4A presents 
State-specific data on the characteristics of near-eligible, eligible non-participating, and participating 
schools in the seven CEP States.) Thi s association could not be examined for New York and Ohio, 
where school ISP data were lacking for some or all non-participating LEAs. In all States except the 
District of Columbia, the percentage of Black students was higher in eligible than near-eligible 
schools, as was the percentage of students eligible for free or reduced price meals (not shown). In all 
States except except Illinois and New York, the average enrollment was lower in participating than 
non-participating schools. 
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Exhibit 4.14: Characteristics of Near-Eligible, Eligible Non-Participating, and Participating 
Schools by State 

Enrollment (average) 

600 - ■- 

iL J . 





ALL DC IL KY Ml NY OH WV 


Identified student percentage 
(average) 


i i i i I ii 


rrrrm 



ALL DC IL KY Ml NY OH WV 


Percent of students who are black 
(average) 


100 



ALL DC IL KY Ml NY OH WV 

■ Near-Eligible ■ Eligible Non-Participating Participating 

Source: State lists of schools with ISPs, Common Core of Data 
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4.2.4 Key Results of Descriptive Analysis 

Across the seven States, a total of 349 regular or charter LEAs and 71 “other” LEAs (mainly non¬ 
public schools) participated in the CEP in SY 2012-13. Within regular or charter LEAs, there were 
3,352 participating schools, as well as 143 schools in “other” types of LEAs (primarily non-public 
schools). Michigan had the most participating regular or charter LEAs and schools in number, but the 
lowest participation rate among all the States. New York had the fewest regular or charter LEAs but 
the most other LEAs. The District of Columbia had the fewest participating schools (but also had the 
fewest eligible schools). Elowever, New York and District of Columbia had the highest participation 
rates for regular and charter LEAs and also for schools in these LEAs. 

Average participation rates in the CEP for SY 2012-13 were 32 percent for eligible regular and 
charter LEAs, and 29 percent for eligible schools in such LEAs. Elowever, participation rates for 
eligible regular and charter LEAs ranged considerably—from 24 percent in Michigan to 65 percent in 
West Virginia. Of note, the Year 1 States and Ohio had participation rates from 24 to 36 percent, 
while the other three Year 2 States had participation rates over 50 percent. This pattern of rates was 
not what one would expect if participation rates increased over time as States gained more CEP 
experience. Data on the number of near-eligible LEAs suggest that the potential to increase the 
number of eligible LEAs was small in most States; only Illinois and Ohio had more than 50 near- 
eligible LEAs. 

Among participating LEAs eligible to participate LEA-wide, a very high proportion of schools 
offered the CEP (97 percent, on average), and 92 percent of these LEAs offered the CEP at all 
schools. On the other hand, among participating LEAs that were not eligible LEA-wide, 79 percent of 
eligible schools offered the CEP, and only 53 percent of these LEAs offered the CEP at all eligible 
schools. 

Participating regular and charter LEAs and schools differed (on average) from eligible non¬ 
participating LEAs and schools in the following ways: 

• Participating LEAs had larger enrollments and numbers of schools, on average, than eligible non¬ 
participating LEAs. However, participating regular and charter LEAs were more often very small 
(with enrollments under 500 students), and participating schools had slightly lower enrollments 
than eligible non-participating schools, on average. 

• Participating LEAs and schools had higher ISPs and FRP meals eligibility percentages, on 
average. 

• Participating LEAs were more often urban, more often either fully or partly comprised of charter 
schools, and had higher percentages of Black students. 

These patterns were replicated in most but not all of the seven States. 

The differences in ISPs and FRP percentages were as expected, given the relationship of the ISP to 
the rate of reimbursement for meals. These factors are generally seen as correlated with urban 
location and concentrations of Black students, and urban LEAs are typically larger and have more 
charter schools. Thus, if ISP is indeed a major determinant of participation, then differences in ISP 
could contribute to the other observed differences. On the other hand, discussions with State Child 
Nutrition staff indicate that the CEP may be more advantageous for charter schools than for regular 
public LEAs, because of the relationship of FRP percentages to educational funding. LEAs use FRP 
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percentages to allocate Federal and State funds among schools. (This consideration is discussed in 
Chapter 3.) 

Given the observational nature of the data, it was not possible to determine which factors were the 
drivers of participation and which were merely correlated. The next section (Section 4.3) presents a 
systematic analysis of the relationships among factors associated with take-up of the CEP. 

4.3 Factors Associated with Take-Up of the CEP 

The previous section reviewed the differences in characteristics of participating and non-participating 
LEAs using descriptive tabulations. This section takes a more systematic approach by using a 
multivariate regression model to understand the specific factors associated with participation in the 
CEP. This approach helps to identify the influence of specific factors while controlling for others. For 
example, Section 4.2 reported that participating LEAs had higher ISPs and FRP percentages. 
Flowever, ISPs and FRPs were highly correlated; that is, LEAs with higher ISPs tended to have 
higher FRP percentages and vice-versa. To separate out the specific association of FRP percentages 
with CEP participation, the ISP (and other variables) must be held constant while varying FRP. The 
multivariate regression analysis presented here does this statistically. As noted in Chapter 2, the 
analysis in this section is descriptive, and therefore not part of the impact analysis, the significance 
tests of factors related to CEP participation are distinct from significance testing of impact estimates 
reported in subsequent chapters. 

4.3.1 Model Specification 

The specific technique used was logistic regression. This technique is appropriate when the dependent 
variable (CEP participation, here) is binary (i.e., the LEA was either a participant or not). The data 
used for this analysis were the same as used in the descriptive analysis presented in the preceding two 
sections, with different LEA inclusion and exclusion criteria. First, LEAs were not excluded from the 
analysis on the basis of CEP eligibility or near-eligibility; instead, all LEAs were included. Second, 
because this model was used to identify matched LEAs for the impact analyses in Chapter 5, a 
number of LEAs were excluded, including 

• All 79 LEAs in the District of Columbia because appropriate matches could not be identified for 
the impact analysis 

• 350 LEAs in Illinois that participated in the Direct Certification with Medicaid (DCM) 
Evaluation 56 

• 523 LEAs across several States serving special needs populations 
Details on and the reasons for these exclusions are discussed in Appendix 2A. 


56 The excluded LEAs were in the treatment group for the DCM Evaluation. These LEAs added Medicaid 
data to the usual SNAP and TANF data used for direct certification. Therefore, it was expected that their 
percentage of students certified for FRP meals, and therefore their rates of student participation in NSLP 
and SBP, would increase. For this reason, these LEAs were excluded from the Impact Evaluation and from 
the analysis of LEA participation in the CEP. 
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The dependent variable in the model was a binary variable equal to one if the LEA had at least one 
CEP participating school in 2012-13 and zero otherwise. Data from all States were pooled, and a 
series of dummy variables for each State was included in the model to capture differences among 
States. The model also included the other variables examined in the previous sections of this chapter 
for their relationship to CEP participation—see Appendix 2B for additional detail on modeling 
decisions. 

• The percentage of identified students (ISP), as a 7-part categorical variable indicating ranges into 
which the LEA’s ISP fell: 0-20, 20-30, 30-40, 40-50, 50-60, 60-70, or 70-100 percent. 57 

• The percentage of students eligible for FRP meals specified as a four-part categorical variable: 

- Less than 50 percent of student eligible for free meals, and less than 10 percent of students 
eligible for reduced price meals; 

- More than 50 percent of student eligible for free meals and less than 10 percent of students 
eligible for reduced price meals; 

- Less than 50 percent of student eligible for free meals, and more than 10 percent of students 
eligible for reduced price meals; and 

- More than 50 percent of student eligible for free meals and more than 10 percent of students 
eligible for reduced price meals. 58 

• The difference between the FRP percentage and the ISP as a measure related to the expected 
difference in reimbursements between conventional meal claiming procedures and CEP 
participation. 

• The percentage of students who were English language learners (ELL) 

• The percentages of students who were in grades K-5, 6-8, and 9-12 (with the percentage in 
grades 9-12 excluded) 

• Whether the LEA was in an urban area (city) 

• The number of schools (natural log) 

• Enrollment (specified as whether the LEA’s average enrollment per school was less than 350, 
which was approximately the sample mean) 

• The percentages of students who were Black and Hispanic/Latino. 

• Percentage of schools that were Title I schools 

• For MI, NY and OH, State-specific indicators for whether the LEA was a charter LEA 


7 This specification was used because exploratory analyses revealed distinct ISP thresholds above which 
CEP participation increased dramatically, and it was important to capture this finding in the results. 

8 This specification was used for two reasons. First, ISP and FRP were very highly correlated. Thus, the two 
could not be included as explanatory variables in their simple forms. Second, the specification allows the 
model to test whether an LEA’s percentages of free and reduced-price eligible students had different 
associations with the CEP participation. See further discussion in Appendix 4A. 
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For the categorical variables, the reference categories were: 

• For ISP, ISP less than 20 

• For FRP, more than 50 percent of student eligible for free meals and more than 10 percent of 
students eligible for reduced price meals 

• For State, Kentucky. 

4.3.2 Results 

The logistic regression results are presented in Exhibit 4.15. Results from logistic regression can be 
presented as “odds ratios,” which are multiplicative factors representing the effect of a variable on the 
probability of an outcome. For example, if the odds ratio for a particular variable was 2, it means that 
a unit increase in that variable was associated with a doubling of the probability of the outcome, 
holding all other factors constant; likewise, an odds ratio of 0.5 means that a unit increase in the 
variable was associated with a halving of probability of the outcome. 

Across all States, there were four statistically significant factors associated with CEP participation: 
the State, the ISP, the number of schools, and the average enrollment in schools. In Ohio alone, 
charter LEAs were significantly associated with greater participation. 

LEAs in Kentucky and West Virginia were much more likely to participate in CEP, all else equal. 
LEAs in Illinois, Michigan, New York, and Ohio were roughly 30 percent—or less—as likely to 
participate in CEP as LEAs in Kentucky, while LEAs in West Virginia were more than twice as likely 
to participate as were LEAs in Kentucky. One possible explanation is based on the fact that LEAs in 
Kentucky and West Virginia are generally aligned with counties, in contrast to other States where 
LEAs covered smaller geographic areas. It was plausible that, being larger and containing more 
schools, LEAs in Kentucky and West Virginia were more likely to have at least one school whose 
participation would be advantageous. An alternate explanation is that differences in CEP 
implementation, such as the role of the State, influenced the LEA participation rate. 

Higher ISPs were strongly associated with a greater likelihood of CEP participation. Increases in ISPs 
from roughly 20 to 60 percent were associated with especially strong jumps in the likelihood of 
participation. All else equal, an LEA with an ISP between 30-40 percent was about five times more 
likely to participate than an LEA with an ISP between 20-30 percent. Further increasing the ISP to 
40-50 percent multiplied the probability of participation by another factor of nearly four. Another 10 
percentage point increase in the ISP to 50-60 percent roughly tripled the probability of participation 
again. Above 60 percent, higher ISPs continued to be associated with a greater likelihood of CEP 
participation, but the effect was less dramatic. 
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Exhibit 4.15: Factors Affecting LEA Participation in the CEP 

Factors 

Odds Ratio 

Significance 

Distribution of ISP indicators 3 

ISP indicator: 20%-30% 

2.20 


ISP indicator: 30%-40% 

11.57 

*** 

ISP indicator: 40%-50% 

45.27 

*** 

ISP indicator: 50%-60% 

135.26 

*** 

ISP indicator: 60%-70% 

181.06 

*** 

ISP indicator: > 70% 

292.44 

*** 

Distributions of free/reduced price meals eligibility 0 

<50% free, <10% reduced price 

1.06 


>50% free, <10% reduced price 

1.37 


<50% free, >10% reduced price 

0.81 


Difference between FRP % and ISP % 

0.20 


LEA characteristics 

Percentage of ELL students 

0.63 


Percentage of students in K-5 grades 

1.31 


Percentage of students in 6-8 grades 

2.24 


LEA in an urban area 

1.03 


Log(Number of schools) 

1.85 

*** 

Avg enrollment per school < 350 

1.56 

*** 

Percentage of African American students 

0.52 


Percentage of Hispanic students 

1.06 


Percentage of Title 1 schools 

0.74 


State charter indicator c 

Ml charter LEA 

1.22 


NY charter LEA 

1.41 


OH charter LEA 

2.72 

** 

State indicators a 

IL LEA 

0.37 

*** 

Ml LEA 

0.31 

*** 

NY LEA 

0.13 

*** 

OH LEA 

0.06 

*** 

WV LEA 

2.51 

*** 

Intercept 

0.01 

*** 


***=p<.01, **=p<.05, *=p<.10. 
a The reference category is ISP below 20%. 

b The reference category is: Greater than 50% of reimbursed meals are free; greater than 10% of reimbursed 
meals are reduced price. 

c Kentucky and West Virginia have no charter schools. In Illinois, charter schools are members in a regular public 
school LEA. 

d The reference category is KY LEA. 

Dependent variable is a dummy variable which takes a value of 1 if the LEA participation in the CEP, and zero 
otherwise. All independent variables are information from previous school year (if available). 

Number of observations (Treatment): 2820 (307). IL: 428(42), KY: 172(49), Ml: 681(106), NY: 673(14), OH: 
811(61), WV: 55(35). 

Interpretation: The odds ratio for ISP of 20-30% is in reference to ISP of less than 20%. So if an LEA with ISP 
between 20-30% is twice more likely to participate in CEP as compared to an LEA with ISP of less than 20%, 
provided everything else remains the same. 

Source: FNS Verification Summary (FNS-742) data, State lists of participating LEAs, Common Core of Data. 
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The effects seem to stem from the reimbursement formula for LEAs participating in the CEP. As 
previously noted, higher ISPs entail higher reimbursement rates. As the LEA ISP approaches 62.5 
percent, the proportion of meals reimbursed at the free rate approaches 100 percent, and thus it 
becomes more advantageous to implement the CEP. 59 

The number of schools serves as a measure of the size of the LEA. According to the results, an LEA 
with 100 schools would be about three times more likely to participate in the CEP than an LEA with 3 
schools, all else equal. The LEA size effect might have occurred for reasons similar to that suggested 
for State effects, namely, LEAs with more schools may have been more likely to have at least one 
school for which participation was advantageous. Other unobserved characteristics associated with 
LEA size, such as the capability of foodservice directors to implement the CEP, might also explain 
this result. The finding that larger LEAs were more likely to participate, all else equal, contradicts the 
suggestion from some States that larger LEAs had more difficulty deciding to implement the CEP. 

The results indicate that LEAs where the average school enrollment was less than 350 were 56 
percent more likely to participate than those where the average school enrollment was more than 350. 
One possible explanation for this is that elementary and middle schools tend to have smaller 
enrollments, and that LEAs with more of these schools (and fewer high schools) may be more likely 
to participate. 60 Another possibility is that smaller schools were more likely to differ from the average 
for the LEA and therefore, at a given level of LEA-wide ISP, an LEA with more small schools was 
more likely to have one or more schools with high ISPs. 

In all States where there were charter LEAs, these LEAs were estimated to be more likely to 
participate in the CEP than their non-charter counterparts. However, only in Ohio was this association 
statistically significant. The results indicate that, compared to other States, Ohio LEAs had the lowest 
overall rate of CEP participation. The charter effect in Ohio significantly boosts this rate for charter 
LEAs, but not enough to raise their participation rates above other States, except New York. 

4.4 Summary 

Across the seven States, a total of 349 regular or charter LEAs and 71 “other” LEAs (mainly non¬ 
public schools) participated in the CEP in SY 2012-13. Within regular or charter LEAs, there were 
2,169 participating schools, as well as 143 schools in “other” types of LEAs (primarily non-public 
schools). Average participation rates in the CEP for SY 2012-13 were 32 percent for eligible regular 
and charter LEAs, and 29 percent for eligible schools in such LEAs. 

Compared with eligible non-participating regular and charter LEAs, participating LEAs of these types 
were larger, on average, had higher ISPs and FRP meals eligibility percentages, had higher 
percentages of students in grades K-5, and had higher percentages of students who were Black. 
Despite their larger average size, participating regular and charter LEAs were more often very small 


59 The percentage of meals reimbursed at the free rate is 1.6 times the ISP. When the LEA’s ISP is 62.5 
percent or greater, all meals are reimbursed at the free rate, and the LEA need not contribute any non- 
Federal funds to make up a difference between the cost of the meals and the Federal reimbursement. 

60 Anecdotal evidence from the State interviews suggests that the impact of the CEP on funding for 
educational programs was more problematic for middle and high schools, as discussed in Chapter 3. 
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(with enrollments under 500 students); they also were more often urban and more often charter 
schools. These patterns were replicated in most (although not all) of the seven States. 

Four factors were significantly associated with CEP participation: ISP, size, State, and charter status. 
The odds of participation rose dramatically with each additional 10 percentage points in the ISP, 
especially between 40 and 70 percent. This result is as expected given how the reimbursement rates 
are determined. The odds of participation also went up with the number of schools in the LEA. 
However, LEAs with below-average-size schools were more likely to participate. The reasons for 
these differences by size are not obvious but may reflect differences in school-level eligibility. LEAs 
in West Virginia were most likely to participate, while LEAs in Kentucky were more likely to 
participate than those in the other four States. Charter LEAs were more likely to participate than 
regular public LEAs, but this finding was significant only in Ohio. Readers should note that the above 
relationships are observed after controlling for the other factors in the analytic model, so they may 
differ from the descriptive results (such as the participation rates by State computed without 
controlling for differences in LEA demographics). 61 Evidence from State interviews suggests that the 
differences by State and charter status may reflect how the CEP was implemented, but differences in 
school-level eligibility also may be contributing factors. 


In addition, the regression analysis used administrative data for all LEAs in six CEP States, while the 
descriptive analysis, which used data only for eligible or near-eligible LEAs, included all seven CEP States 
as of SY 2012-13. 
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5. Participation and Revenue Impacts 


This chapter assesses the effects of the Community Eligibility Provision (CEP) on program 

participation and revenues, and addresses the following research questions: 

• How did average daily participation in the NSLP change because of the CEP (compared to what 
participation would have been in the absence of the CEP)? 62 

• How did average daily participation in the SBP change because of the CEP (compared to what 
participation would have been in the absence of the CEP)? 

• To what extent (if any) did the impacts of the CEP on program participation reflect changes in the 
number of actual participants, frequency of participation, or both for NSLP, SBP, and total 
meals? 

• To what extent (if any) did the impacts of the CEP on program participation vary by grade level 
for NSLP, SBP, and total meals? 

This chapter also addresses the following research questions about the impact of CEP on LEA 

foodservice revenues: 

• How did Federal revenues (i.e., LEA revenues from USD A reimbursements) per reimbursable 
meal for NSLP change because of the CEP (compared to what revenues would have been in the 
absence of the CEP)? 

• How did Federal revenues per reimbursable meal for SBP change because of the CEP (compared 
to what revenues would have been in the absence of the CEO)? 

• How did the total Federal revenues per student change because of the CEP (compared to what 
revenues would have been in the absence of the CEP)? 63 

• How did revenues from non-Federal sources, including State funds, local funds, student payments 
for reimbursable meals, and payments for competitive foods 64 and adult foods, change because of 
the CE Provision compared to revenues that would be generated from traditional meal 
reimbursement structure? 


62 Because of the multiple comparisons issue, the evaluation identified a single measure as confirmatory 
within the participation outcome domain. Candidate measures included NSLP participation, SBP 
participation and total participation. The NSLP was selected for two reasons: the NSLP is the larger 
program, and the estimates of the effect of the CEP on NSLP participation were less likely to be biased. 
The presence of the SBP program at a school is potentially related to a school’s choice to take lip the CEP, 
and so disentangling the selection effect from the causal effect is more difficult for the SBP. Therefore, the 
impact on SBP participation is better suited for an exploratory analysis. 

63 As discussed in Section 5.2, total Federal revenues were normalized on a per student basis to permit 
comparisons between LEAs of different sizes. 

64 The term “competitive foods” refers to all foods sold to students by school foodservice other than 
reimbursable meals. As such this does not include food sales by other entities, such as the athletic 
department or school store. 
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Finally, the chapter addresses the impacts of the CEP on the availability and operation of the SBP in 
schools. One of the requirements of the CEP is that all schools must participate in the School 
Breakfast Program (SBP). Implementation of the CEP may have led some LEAs to begin offering 
breakfast, if the LEA had not been doing so previously. Also, since the CEP also requires that all 
reimbursable meals be served free to all students, the CEP may have prompted some LEAs to vary 
their approach to breakfast service. The research questions in this domain were the following: 

• How did the CEP impact the SBP in terms of whether schools offer breakfast and the type of 
breakfast service offered? 

• In LEAs and schools with SBP already in place, what alternative types of breakfast service were 
tried in lieu of the traditional cafeteria setting? Which of these have been retained as ongoing 
routine practices? 

• To what extent did the CEP affect the foods served at breakfast, specifically whether the identical 
breakfast is served to all students? 

• How much time are students allowed to eat breakfast? 

The questions of impact on NSLP participation and Federal revenues for the NSLP were identified in 
advance as the study’s two confirmatory research questions, i.e., the questions by which the 
effectiveness of the CEP will be judged. The primary goal of the CEP is to increase access to and 
participation in the NSLP and SBP among students in high-poverty schools. The remaining questions 
about participation, revenues, and availability and operation of the SBP were identified as 
exploratory: they are questions of interest, and the results are not, therefore, considered conclusive. 

As discussed in Chapter 2, drawing confirmatory conclusions on additional research questions would 
require adjustments for multiple comparisons, and as a result reduce the power to detect any impacts. 

Analyses of the impact of the CEP on NLSP and SBP participation and revenues used State 
administrative data and a comparative interrupted time series design (CITS) approach to compare 
what happened in CEP-participating LEAs (the treatment group) to a counterfactual—what would 
have happened in the program’s absence. First, a comparison group of LEAs similar to the treatment 
group of LEAs on pre-participation characteristics was constructed using propensity score matching; 
next, the difference in the pre- and post-CEP levels of program participation (or revenue) was 
compared to the corresponding pre-post differences in the matched comparison group. (See Chapter 2 
for further description of the study design.) 

The other analyses presented in this chapter used data from Participation, Enrollment, Attendance, 
and Revenue (PEAR) survey respondents from CEP-participating schools and comparison schools. 
For revenue data, the CITS approach was used. For impacts on the availability and operation of the 
SBP, the analysis used a posttest-only comparison of the treatment LEAs and their matched 
comparison LEAs. Supplementary data on the operation of the SBP were drawn from the 
Implementation Web Survey of Participating LEAs and the Pre-visit Questionnaire administered as 
part of the Menu Survey 

Before presenting the results for the impact of the CEP on participation, revenues, and SBP 
availability and operations, this chapter first (1) discusses expectations about why and how effects 
might arise; (2) presents the specific outcome measures used; (3) describes in more detail the 
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methodology for selecting the comparison group for analysis of participation and revenue; and (4) 
details the models used for analysis. 

5.1 Expectations about CEP Impacts on School Meals Participation, 
Revenues, and SBP Availability and Operations 

The research questions were motivated by a conceptual model of how the CEP might be expected to 
affect NSLP and SBP participation rates and revenues, and SBP availability and operations. Adoption 
of the CEP by schools could lead to increased NSLP and SBP participation for several reasons. First, 
any schools that had not had a breakfast program were required to add one (and thus increasing the 
availability of the SBP would result in additional participation). Second, the CEP made it easier for 
students eligible for free meals to participate by eliminating the burden of the application process. 
Third, in CEP schools, meals were now free for those students who otherwise would have had to pay 
for reimbursable meals. Fourth, universal availability may have reduced the stigma associated with 
taking free meals. Finally, schools may have made efforts to promote universally available meals or 
to improve meal delivery resulting in increased participation. 

On the revenue side, under the CEP, the LEA receives higher USD A reimbursements for meals 
served to students whose incomes exceed the eligibility levels for free meals. Elowever, the LEA 
forgoes the revenue from student payments for reduced-price (RP) and paid meals (i.e., because these 
students now receive meals for free). Unless the LEA has a high enough ISP (62.5 percent or more) 
that it claims all meals in CEP schools at the free meals rate, it receives less Federal revenue for a 
meal served to a free-eligible student under the CEP than it would under conventional claiming 
procedures. 65 Under conventional reimbursements, an LEA’s total revenues for a meal served to a 
student eligible for RP meals equals USDA reduced-price rate plus the student payment, which is set 
at the difference between the USDA rates for free and RP meals. Thus, only LEAs that claim all 
meals at the free rate under the CEP get the same total revenue per meal served to RP-eligible 
students as under conventional procedures. Whether the LEA gains or loses revenues on meals that 
would otherwise be served at the paid rate depends upon both the price charged as well as the ISP. 66 

The CEP also is likely to affect LEA foodservice revenues from non-Federal sources. The impact on 
revenues from State funding depends on how the State provides assistance to LEAs: if the State 
provides reimbursements tied to counts of free, reduced-price, and paid meals, then the impact will 
depend on the ISP (just as it does for Federal revenues). Revenues from non-reimbursable foods 
might fall (if students choose free reimbursable meals over paying for non-reimbursable foods), or 
rise (if students buy more non-reimbursable foods to supplement their free reimbursable meals). 

LEAs that have a net loss in revenues from Federal, State, and student payments may need to increase 


65 As discussed in Chapter 1, the ISP is multiplied by 1.6 to determine the claiming percentage, which is the 
percentage of meals that were reimbursed at the free meals rate, which was $2.86 in SY 2012-13 (unless 
the LEA qualified for higher rates due to high need or certification for meeting school meals standards). 
Under conventional claiming procedures, the LEA received $2.86 in SY 2012-13 for each meal served to a 
free-eligible student. Under the CEP, the reimbursement rate for all meals was $2.86 if (1.6*ISP)=1 or 
ISP=1/1.6=0.625. 

66 For example, if the LEA charges $2.00 for a paid lunch and gets $0.27 from USDA, the total revenue is 
equivalent to the CEP reimbursement rate of $2.27 per lunch when the ISP equals 49.6 percent. So if the 
ISP is higher, the LEA would receive more total revenue per lunch. 
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the amount of local funds supporting school foodservice, unless they are able to reduce costs enough 
that they still balance their school foodservice accounts. 

As discussed above, the expected impact of the CEP on availability of the SBP is clear: schools that 
do not already offer the SBP must do so to participate in the CEP. Regarding SBP operations, schools 
may use new ways of delivering breakfasts under the CEP, such as “grab and go” methods or serving 
in the classroom, to accommodate and facilitate greater participation, and also take advantage of 
eliminating the need to collect payments. These choices, or cost considerations, may affect the variety 
of foods offered at breakfast or the time students have to eat breakfast. 


5.2 Outcome Measures 

5.2.1 Participation 

Participation in the NSLP and SBP was measured as the percent of students who receive meals. These 
outcomes were operationalized as follows: 


Average Daily Participation (ADP) for NSLP: 


ADP NSL p 


# of NSLP monthly meals 1 

# of operating days in the month enrollment 


(Eq. 5.1) 


Average Daily Participation (ADP) for SBP: 


ADP 


SBP 


# of SBP monthly meals 1 

# of operating days in the month enrollment 


(Eq. 5.2) 


5.2.2 Federal Revenue 


Federal revenue from NSLP and SBP was measured as the Federal reimbursement rate per meal 
served. This measure reflects the Federal dollar amount received for each meal served by the LEA 
during the school year. These rates were operationalized as follows: 


Federal Reimbursement per NSLP meal: 


Federal reimbursement rate NSLP 


$ NSLP monthly Federal revenue 
# NSLP monthly lunches served 


(Eq. 5.3) 


Federal Reimbursement per SBP meal: 


Federal reimbursement rate SBP 


$ SBP monthly Federal revenue 
# SBP monthly breakfasts served 


(Eq. 5.4) 


In addition, the revenue was also measured as the total Federal revenue received by the LEA for each 
student enrolled. This measure reflects the average monthly Federal dollar amount received for each 
student served by the LEA during the school year. Normalizing total revenue on a per-student basis 
permitted comparisons among LEAs of different size and over time. This outcome reflects both the 
rate of participation in NSLP and SBP and the reimbursement per meal. It was operationalized as 
follows: 


Total Federal Revenue per Student 


$ NSLP monthly rev.+ $ SBP monthly rev.+ $Afterschool Snack monthly rev. 
enrollment 


(Eq. 5.5) 
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The data source for the construction of these outcomes was the State-provided administrative data, 
which included NSLP and SBP monthly meal counts; monthly Federal revenue for NSLP, SBP and 
Afterschool snack program; enrollment; and the number of operating days in the month. 

5.2.3 Non-Federal Revenues 

The analyses of non-Federal revenues assessed the impact of the CEP on revenues from: (1) State 
funds; (2) local funds; (3) student payments for non-reimbursable meals; and (4) payments for 
reimbursable foods. These analyses combined data from the Participation, Enrollment, Attendance 
and Revenue (PEAR) survey with State Administrative Data. Each of the sources was treated slightly 
differently based on data availability and the nature of the revenue. 

(1) Revenues from State funds were measured as the total fall revenue received from State 
funds per student among LEAs that report receiving State Revenue. 67 This was 
operationalized as: 

. r, $ Jul.State rev.+ $ Aug.State Rev.+ $Sep.State rev.+ $Oct.State rev.+ $Nov.State rev.+ $Dec.State rev. c r' 

State Revenue = —- 5 --- (Eq. 5.6 

enrollment v 1 


where enrollment was obtained from State Administrative Data. State revenues were obtained from 
the PEAR survey for all States other than Illinois and from State Administrative Data for Illinois. 68 


(2) Revenue from local funds was measured as a binary variable that indicates whether the 
foodservice program received funding from local sources (i.e., a transfer of funds to the 
school foodservice account from another LEA account) 69 in a given school year, 
operationalized as: 


Local Revenue = 


G 


no funds were transferred from other LEA account to school foodservice 
funds were transferred from other LEA account to school foodservice 


(Eq. 5.7) 


This definition excludes transfers to pay for catering for local events. 70 The evaluation team selected 
this definition of the outcome to capture the key variation among LEAs in terms of whether local 
funds were used for foodservice, and to maximize the number of LEAs included in the analysis (see 
Appendix 5A for details). These data were obtained via the PEAR survey for all States. 


(3) Revenue from Payments for Non-Reimbursable Foods was measured as a binary variable 
that indicates whether the foodservice program received payments for non-reimbursable 
foods, including competitive foods, a la carte, vending machines or adult meals. Thi s variable 
was defined as: 


67 The fall total was used rather than school year total to insure consistency of measurement across years. The 
timing of the data collection was such that the evaluation team was only able to collect data on revenues 
from July to December of 2012. 

68 See Appendix 5A for additional details on outcome construction and the definition of the analytic sample. 

69 If the school foodservice account has a deficit (i.e., costs exceed revenues), the LEA typically transfers 
funds from the LEA’s general fund to balance the foodservice account. The general fund is the fund for all 
revenues that are not restricted to specific programs or grants. Under the CEP, LEAs must use non-Federal 
funds to cover any costs of the NSLP and SBP that are not covered by Federal reimbursements. 

70 Payments for catering services were captured in a separate revenue category on the PEAR survey. 
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NonReimb Revenue = 


no revenue generated from competitive foods, a la carte, vending machines, or adult meals 
revenue generated from competitive foods, a la carte, vending machines, or adult meals 


(Eq. 5.8) 


As was the case for local revenues, the evaluation team selected this definition of the outcome to 
capture the key variation among LEAs in terms of whether revenue was generated from payments for 
non-reimbursable meals and to maximize the number of LEAs included in the analysis. These data 
were obtained from the PEAR survey for all States other than Kentucky. 71 See Appendix 5A for 
additional detail on the definition of the outcome variable, data availability and the definition of the 
analytic sample. 


(4) Revenues from Student Payments for Reimbursable Meals were measured by the fall total 
revenue received from student payments per student for LEAs that report such revenue and is 
defined to be zero for LEAs that report that no revenue was generated from student payments. 
This is operationalized as: 


Student Revenue = 


'Fall Student Revenue 
. 0 


if Fall Student Revenue was reported 
if LEA indicated no revenue was generated by student payments 


(Eq. 5.9) 


where 


_ „ _ , _ $Jul.Studentrev.+ $Aug.StudentRev.+ $Sep.Studentrev.+ $Oct.Studentrev.+ $Nov.Studentrev.+ $Dec.Studentrev. 

Fait Student Revenue = --- 

enrollment 


(Eq. 5.10) 


This outcome measure was defined for all LEAs that reported not having generated revenue from 
student payments, and for all LEAs that reported the fall revenue from student payments. The student 
revenue outcome measure was not available for LEAs that indicated that revenue was generated from 
student payments, and that did not report fall student revenue, either because the LEA was unable to 
report student revenue separately or because the fall student revenue data were not available. 72 On 
average, this outcome measure tends to understate the revenue generated from Student Payments for 
Reimbursable Meals, because it includes all LEAs indicating that no such revenue was generated, but 
not all LEAs indicating that revenue was generated. 


This understatement is expected to be more pronounced for LEAs that would be expected to have 
student payments for reimbursable meals (including comparison LEAs and treatment LEAs in pre¬ 
intervention years) than for LEAs actively implementing the CEP in that year. Schools implementing 
the CEP were required to offer reimbursable meals free to all students. Therefore, LEAs that offered 
the CEP in all schools should have indicated that no student revenue was generated by student 


71 The PEAR survey was customized for each State based on the revenue data available from the State, so that 
LEAs were only asked to provide data that the State did not have. For Kentucky, the PEAR survey did not 
request data on revenues from non-reimbursable meals and student payments for reimbursable meals 
because it was expected that the State would provide the data. However, the data were not available from 
the State for the fall of 2012-13. Therefore, Kentucky was excluded from the analyses of these outcomes. 

72 See Exhibit 5A.23 for the number of LEAs that reported generating no student revenue, the number that 
indicated generating student revenue but were unable to report fall student revenue, and the number that 
indicated generating student revenue and reported fall student revenue. 
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payments in the years the LEA offered the CEP and been correctly represented in the data as having 
$0.00 in revenue from student payments for reimbursable meals. 73 

Data on student payments for reimbursable meals were obtained from the PEAR survey for all States 
other than Kentucky (as explained on the preceding page). 

5.3 Constructing Matched Comparison Groups 

The evaluation design for these analyses was based on using both longitudinal (time series) and 
comparison group data to isolate the effect of the CEP on relevant outcomes, and was designed to 
eliminate many sources of bias from those impact estimates. This design was used for analyses of two 
different samples based on available data: estimating the impact of CEP on Participation and on 
Federal Revenues using the Matched Administrative Data Sample; and estimating the impact of CEP 
on non-Federal Revenues using the Component 1 PEAR Survey Sample. For both samples, 
participating LEAs were identified as the treatment group, as is usual practice in discussing a quasi- 
experimental design. To select a comparison group of non-participating LEAs, propensity score 
matching (PSM) was used. PSM was intended to construct treatment and comparison groups that 
were equivalent at baseline, using pre-program characteristics of the LEAs—as in the analysis of 
factors associated with CEP take-up. 

5.3.1 Comparison Group Matching for Analyses of Participation and Federal Revenues 

The propensity score matching used pre-CEP characteristics data from States as well as the Co mm on 
Core of Data (CCD), and FNS-provided administrative data (Verification Summary Report (VSR), 
the same data source used to identify factors that influenced take-up of the CEP (see Chapter 4 for 
these characteristics and Appendix 4 for added detail). The PSM technique used resulted in up to five 
potential matches for each LEA in the treatment group 74 ' 75 (see Appendix 4A for a more detailed 
analysis of the matching process). 


73 The implications of the overrepresentation of LEAs that generate no revenue from student payments for 
reimbursable meals for the analysis are discussed in the section presenting the results of the analysis. See 
Appendix 5 for additional detail on the data availability and the definition of the analytic sample. 

74 A one-to-five nearest neighbor with caliper matching strategy was used to obtain the comparison group of 
LEAs. This matching procedure allows up to five non-participating LEA to provide information on the 
counterfactual scenario for each participating LEA. Further, each non-participating LEA is allowed to be 
matched to an unlimited number of participating LEA. The matching procedure constructs weights for 
comparison LEAs that reflect the number of LEAs for which it was selected to serve as comparison and 
how many comparisons were identified for each of those LEAs. To prevent the use of non-comparable 
LEAs as matches, the evaluation team required the treatment and comparison LEAs to have propensity 
scores within 1 percentage point of one another. 

73 If the PSM model predicts take-up well, then participating LEAs will have relatively high scores, and 

matches from non-participating LEAs with similarly high scores will be chosen to comprise the comparison 
group. If there is greater dispersion of the propensity score across participating LEAs than across non¬ 
participating LEAs, then there is less predictive accuracy; however, the process still chooses matches from 
among non-participating LEAs that have scores similar to those selected for the treatment group. There was 
considerable overlap between the propensity scores of the treatment group and the comparison group. 


Abt Associates 


5. Participation and Revenue Impacts | pg. 97 






Community Eligibility Provision Evaluation: Final Report 


The samples for analyses of NSLP participation, NSLP revenue, and total Federal revenue were 
identical to the matched administrative sample used for analysis of CEP take-up. Exhibit 2.6 shows 
the characteristics of the sample treatment and comparison LEAs. The matched sample for SBP 
differed due to some missing information. Exhibits 5A. 1 and 5A.2 in Appendix 5A show the results 
of baseline balance testing for participation and revenue sample for SBP, respectively. 

For both NSLP and SBP participation and revenue, across nearly all pre-CEP characteristics, there 
were no differences between the treatment and comparison LEAs . There was a small difference with 
respect to average daily student participation in the SBP; treatment LEAs had a slightly higher rate, 
on average, than comparison LEAs (47 percent versus 42 percent). 76 Regardless, the two groups 
appear to be quite well-matched on the observed characteristics that the analyses used to identify the 
comparison LEAs. 

5.3.2 Comparison Group Matching for Analyses of Non-Federal Revenues 

For the Component 1 PEAR Survey Sample, the comparison group was identified using a different 
PSM approach based on State administrative, VSR and CCD data available prior to data collection 
(see Appendix 2A for details). Propensity score modeling was performed separately for each State to 
support within-State matching; this resulted in a single comparison LEA matched to each treatment 
LEA. 

The set of LEAs included in each of the analyses of non-Federal Revenues had unresolvable 
imbalances on several factors (i.e., differences at baseline) between the treatment and comparison 
groups. 77 (See Exhibit 5 A. 10 in Appendix 5A for statistics.) The combination of several factors— 
specifically survey non-response, item-level non-response and after-the-fact identification of LEAs as 
ineligible based on survey responses—resulted in substantial differences between the initial PEAR 
survey sample and the final analytic sample of LEAs for analyses of non-Federal revenues. 78 The 
analytic sample for the State revenue analysis was imbalanced on several baseline characteristics: 

ISP, percent of Title I schools, charter status and the percent of students who are Hispanic; likewise, 
the analytic sample for the analysis of revenue from student payments for reimbursable meals was 
imbalanced on baseline ISP and percent Title I schools. The analytic model for each of these two 
analyses included baseline covariates on which the sample was imbalanced. The analytic samples for 


However, the treatment group had propensity scores closer to one, and finding suitable matches for these 
LEAs was difficult. 

76 A possible reason for treatment-comparison differences on this variable is that it was not part of the 
propensity score model. All variables included in the model are balanced between the two groups. 

77 Exhibit 2.7 in Chapter 2 presents the baseline balance tests for the overall sample of PEAR survey 
(Component 1) respondents. However, because of logical skips in PEAR survey programming and item- 
level non-response, not all of the PEAR survey respondents provided sufficient data for inclusion in the 
analysis of non-Federal revenue. The baseline balance tests presented in Appendix 5 A restrict attention to 
the analytic sample for each of these analyses. 

78 Note that propensity score matching was performed prior to data collection. All PEAR survey respondents 
with sufficient data were included in each analysis. Baseline balance testing results capture the extent to 
which the treatment and comparison groups are comparable in the resulting analytic sample. 
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the local revenue and revenue from payments for non-reimbursable foods were each balanced on 
baseline characteristics. 


5.4 Analytic Model 

This section presents the analytic model used to estimate the CEP’s impact on participation, Federal 
revenues and non-Federal revenues. Specifically, it describes how the time series data were analyzed. 
The basic model for estimating impacts is shown in equation 5.11: 

Y it = (00 + a o i) + (01 + «ii)t + 02 Ti + 03 %t + 04 t TEit + 05 z i + £ it (1) (Eq. 5.11) 

Where 

Yit is the outcome of interest (participation, revenues) for LEA i in time t; 

7) is the treatment group indicator (=1 if LEA i takes the CEP at any point in time; =0 otherwise); 

X t is a post-treatment indicator (=1 if in the years the CEP is available and 0 otherwise); 

TE it is the treatment effect indicator (=1 if LEA i implemented the CEP at time t; =0 otherwise); 

Z[ is a vector of pre-intervention characteristics of LEA i (this is an LEA-level measure—not 
time varying—during the pre-intervention years), a oi is the between-LEA random variation in the 
intercept (deviation of LEA V s intercept from the grand mean intercept, distributed with mean 0 
and variance Tq); 

a lt is the between-LEA random variation in the slope (deviation of LEA i’s slope from the grand 
mean slope, distributed with mean 0 and variance rf); and 

£ it is the error term, which can be further decomposed into an LEA-specific unobservable 
constant term and an independent, identically distributed error term u it . 

In this model, the effect of participating in CEP is given by 0 4 , the average treatment effect. This 
coefficient corresponds to the difference between the pre- and post-outcomes observed for the 
treatment LEAs and those observed for the comparison LEAs. The coefficients 0 O and 0! are 
interpreted as the regression-adjusted level and change over time in the mean outcome for comparison 
LEAs. 0 O + 02 and 0 X provide the regression-adjusted level and change over time of the mean 
outcome among treatment LEAs. 0 3 captures the influences of policy changes other than the CEP on 
the outcome, and 0 5 captures the contributions of other explanatory variables. 

Because non-Federal Revenue sources did not show evidence of a time trend in the pre-intervention 
period, time was omitted from the model for these outcomes. This corresponds to setting 0 4 and a x to 
equal 0 in Equation 5.11 above. 

5.5 Impact of CEP on Participation in School Meal Programs 

This section presents the impact of the CEP on NSLP and SBP participation rates, based on whether 
these rates differed for those LEAs that did and did not participate in the CEP. This analysis used the 
LEA-level CITS analysis described Section 5.4. See Appendix 5A for further information on the 
CITS analysis model used. 
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The impact of the CEP on participating LEAs was examined by comparing the participation rates in 
NSLP before and after the intervention to the participation rates during the same time periods for a 
matched group of non-participating LEAs. Exhibit 5.1 displays the regression-adjusted means for the 
treatment and comparison LEAs. The average daily participation rates in NSLP for the treatment 
LEAs increased to 76.3 percent from 72.4 percent, whereas the matched comparison LEAs’ rates held 
nearly constant, changing from 68.5 percent to 68.9 percent over the same time. The change in the 
comparison LEAs represents the change that would have occurred in the treatment LEAs in the 
absence of CEP participation. Thus, subtracting out the 0.4 percent change experienced by the 
comparison group from the 4.0 percent change experienced by the treatment groups produces the 
impact of the CEP: a 3.5 percentage point increase in school meal program participation. 79 This 
impact is statistically significant at the one percent level. As an indicator of the relative magnitude of 
the impact, the NSLP participation rate in CEP LEAs was 5.2 percent higher than expected in the 
absence of the program (as represented by the comparison group’s pre-intervention mean 
participation rate). Results from the full model (explained in Section 5.4) appear in Appendix 5A. 


Exhibit 5.1: Impact on NSLP Participation Rate 






Percent 


Comparison 

Treatment 

Impact 

Impact 

Mean Pre-Intervention Average 
Daily Participation (ADP) Rate 

68.5% 

72.4% 



Mean Post-Intervention ADP Rate 

68.9% 

76.3% 



Difference = (Post-Intervention 

0.42 

percentage pts 

3.96 

percentage pts 

3.54 


ADP—Pre-Intervention ADP) 

percentage 

pts*** 

5.2% 

Number of LEAs in sample 

525 

286 




For the purposes of this table, the average treatment effects are for Year 1 States LEAs that adopted the 
intervention in SY 2011-12. The Average Pre-Intervention ADP is the average of the regression-adjusted ADP 
for 2009 and 2010. The Average Post-Intervention ADP is the average of the regression-adjusted ADP for 2011 
and 2012. The pre- and post-intervention time periods for LEAs that implemented the CEP in 2012 are different, 
and so the regression-adjusted means are different, but the estimated impact is the same because it derives 
from the same parameter in the model. 

***=p<0.01, **=p<0.05, *=p<0.10. 

Source: State-provided administrative data. 

The same methods were used to estimate the CEP’s impact on participation in the SBP. Exhibit 5.2 
displays the regression-adjusted means for the participating (treatment) LEAs and non-participating 
(comparison) LEAs. 80 The adjusted average daily participation rates in SBP for the treatment LEAs 
increased from 46.2 percent to 52.3 percent while the matched comparison LEAs’ participation rate 
increased from 38.3 percent to 40.7 percent over the same time. Taking the difference between these 
two differences yields the CEP’s impact on SBP participation: 3.6 percentage points. This impact is 


79 The estimated impact of 3.5 percent differs from the computed difference between the treatment and 
control group differences (4.0 percent -0.4 percent) due to rounding. The more precise estimates are: 
treatment group difference, 3.96 percent; comparison group difference, 0.42 percent; difference in 
differences, 3.54 percent. 

80 As in Exhibit 5.1, estimates are for LEAs that implemented the CEP in SY 2011-12. 
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also statistically significant, and represents a 9.4 percent increase in the participation rate for the SBP. 
Results from the full model (explained in Section 5.4) appear in Appendix 5A. 


Exhibit 5.2: Impact on SBP Participation Rate 






Percent 


Comparison 

Treatment 

Impact 

Impact 

Average Pre-Intervention ADP 

38.3% 

46.2% 



Average Post-Intervention ADP 

40.7% 

52.3% 



Difference = (Post-Intervention 
ADP - Pre-Intervention ADP) 

2.4pts 

6. Opts 

3.6pts*** 

9.4% 

Number of LEAs 

520 

286 



Number of observations 

1,995 

1,124 




For the purposes of this table, the average treatment effects are for Year 1 States LEAs who adopted the 
intervention in SY 2011-12. The Average Pre-Intervention ADP is the average of the regression-adjusted ADP 
for 2009 and 2010. The Average Post-Intervention ADP is the average of the regression-adjusted ADP for 2011 
and 2012. The pre- and post-intervention time periods for LEAs that implemented the CEP in 2012 are different, 
and so the regression-adjusted means are different, but the estimated impact is the same because it derives 
from the same parameter in the model. 

***=p<0.01, **=p<0.05, *=p<0.10. 

Source: State-provided administrative data.. 

Results from the PEAR survey provide information for addressing the two exploratory questions 
regarding possible reasons for the observed impacts on CEP participation. Seventy-one percent of the 
participating LEAs perceived that their ADP for the SBP had increased since January 2011, compared 
to 45 percent of non-participating LEAs. Similarly, more than three-fifths of the treatment LEAs 
perceived an increase in their ADP for the NSLP, compared to one-quarter of the comparison LEAs 
(see Exhibit 5.3). Among LEAs that reported an increase in their ADP (both treatment and 
comparison LEAs), the most commonly cited factors for this increase were a greater percentage of 
students participating in the program (about 60 percent of LEAs cited this) and a greater number of 
meals taken per week, for both SBP and NSLP (see Exhibit 5.4). Additionally, about 50 percent of the 
LEAs that reported an increase in their SBP’s ADP identified elementary schools in their LEA as the 
having the highest increases in ADP rates (see Exhibit 5.5). 

The few treatment LEAs that reported decreases in their ADP attributed that decrease to different 
factors for the NSLP and the SBP, respectively. For the NSLP, they attributed the decrease in ADP to 
a decline in student enrollment, 81 but for the SBP, they attributed the decrease to a decline in the 
percent of students participating in the program (see Exhibit 5.6). In contrast, among the comparison 
LEAs that reported ADP decreases, more than three-fourths attributed these decreases to a 
combination of a smaller percentage of students participating in school meal programs and fewer 
meals taken per week (for both SBP and NSLP). Among those reporting decreases, a plurality of both 
treatment and comparison LEAs identified high schools as having the highest decreases in ADPs (see 
Exhibit 5.7). 


81 LEAs attributing a decline in ADP (a per-student measure) to declining enrollment may have been 

confused by the question and intended to explain a decline in total meals served. Due to the small number 
of respondents indicating declines in ADP, this pattern was considered a minor anomaly. 
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Exhibit 5.3: Percentage of Participating and Non-Participating LEAs Reporting Changes in 
ADP for SBP and NSLP 


Perceived change in ADP 
in the program since 
January 2011 

Breakfast 

Lunch 

Comparison 

Treatment 

Comparison 

Treatment 

Increased ADP 

45% 

71%*** 

25% 

61%*** 

Decreased ADP 

15% 

11% 

35% 

16%*** 

Stayed the same 

40% 

17%*** 

40% 

23%“* 

Total 

95 

133 

96 

134 


Percentages reported in round numbers due to small sample size. 
***=p<0.01, **=p<0.05, *=p<0.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey. 


Exhibit 5.4: Percentage of Participating and Non-Participating LEAs Citing Reasons for a 
Reported Increase in ADP for SBP and NSLP 


Biggest factor in perceived 

Breakfast 

Lunch 


increase in ADP 

Comparison 

Treatment 

Comparison 

Treatment 

More schools offering school 
breakfast 

5% 

1% 



An increase in student 
enrollment 

10% 

14% 

17% 

17% 

An increase in the 
percentage of students 
participating 

59% 

60% 

61% 

54% 

An increase in the number of 
meals per week taken 

27% 

24% 

22% 

28% 

Total 

41 

91 

23 

81 


Percentages reported in round numbers due to small sample size. 
***=p<0.01, **=p<0.05, *=p<0.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey. 


Exhibit 5.5: Percentage of Participating and Non-Participating LEAs Attributing a Reported 
Increase in ADP for SBP and NSLP to Particular School Level 


Type of school with 
greatest perceived 
increase in ADP 

Breakfast 

Lunch 

Comparison 

Treatment 

Comparison 

Treatment 

Elementary school 

46% 

52% 

40% 

44% 

Middle school 

8% 

7% 

0% 

8%** 

High school 

14% 

21% 

10% 

19% 

No difference by type 

32% 

21% 

50% 

29%* 

Total 

37 

92 

20 

79 


Percentages reported in round numbers due to small sample size. 
***=p<0.01, **=p<0.05, *=p<0.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey. 
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Exhibit 5.6: Percentage of Participating and non-Participating LEAs Citing Reasons for a 
Reported Decrease in ADP for SBP and NSLP 

Biggest factor in perceived 

Breakfast 

Lunch 


decrease in ADP 

Comparison 

Treatment 

Comparison 

Treatment 

A decrease in student 

enrollment 

18% 

64%*** 

24% 

38% 

A decrease in the percentage 
of students participating 

45% 

29% 

30% 

52% 

A decrease in the number of 
meals per week taken 

36% 

7%* 

45% 

10%*** 

Total 

11 

14 

33 

21 


Percentages reported in round numbers due to small sample size. 
***=p<0.01, **=p<0.05, *=p<0.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey. 


Exhibit 5.7: Percentage of Participating and Non-Participating LEAs Attributing a Reported 
Decrease in ADP for SBP and NSLP to Particular School Level 


Type of school with 
greatest perceived 
decrease in ADP 

Breakfast 

Lunch 

Comparison 

Treatment 

Comparison 

Treatment 

Elementary school 

15% 

8% 

12% 

11% 

Middle school 

15% 

15% 

15% 

0%** 

High school 

38% 

31% 

38% 

47% 

No difference by type 

31% 

46% 

35% 

42% 

Total 

13 

13 

34 

19 


Percentages reported in round numbers due to small sample size. 
***=p<0.01, **=p<0.05, *=p<0.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey. 


5.5.1 Discussion of Participation Results 

The CEP’s impact on student participation in school meal programs was surprisingly consistent, for 
both programs, increasing participation by 5 to 9 percent (about 3.5 percentage points for the NSLP 
and SBP participation). 82 Nevertheless, the estimates produced by this analysis must be interpreted 
carefully. These estimates, at best, measure the effect of the CEP on the specific group of LEAs that 
were observed: those that chose to participate in the first two years of availability in the Year 1 States 
(Illinois, Kentucky and Michigan) and those that chose to participate in the first year of availability in 
the three Year 2 States (New York, Ohio, and West Virginia). These estimates do not generalize to 
LEAs in these States that are not comparable to the participating LEAs, nor do they generalize to 
LEAs in other States. 

The extent to which the study’s analyses effectively identify the true impact of adopting the CEP 
depends on how well the experiences of the untreated comparison group represent what would have 
happened to schools in the treatment group had they not chosen to participate in CEP. In this case, 
selection bias may result from the fact that LEAs chose the schools to implement the CEP. When 
making this decision, presumably the LEA considered whether participating or not would produce 


These findings are robust to alternative model specifications presented in Appendix 5A. 
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better outcomes for the schools within the LEA. Therefore, it can be expected that schools within a 
given LEA that did participate likely differed systematically from schools in the LEA that did not 
participate in ways that directly affect the outcomes of interest. Moreover, the propensity score 
matching approach identified comparison LEAs using characteristics that were measureable and 
easily accessible. These characteristics do not account for other, potentially important, unmeasurable 
characteristics that determine the take-up decisions of LEAs, such as the perception that the LEA will 
benefit from adopting the CEP. On the other hand, given that in the average participating LEA, 94 
percent of eligible schools participated in CEP, the effect of differences between schools within the 
LEA may be modest. 83 In addition, the propensity score matching and comparative interrupted time 
series (CITS) approach provided the most rigorous and feasible way to separate the impact of the 
CEP from the other factors that could have affected NSLP and SBP participation in the sample. 
Moreover, additional tests showing the robustness of the results (see Appendix 5A) provides 
additional confidence in the conclusion that the CEP did indeed increase NSLP and SBP 
participation, as do the PEAR survey results. Both of the estimated relative impacts (5.2 percent and 
9.4 percent) are of sufficient magnitude that they are likely to be seen as not only statistically 
significant but also substantively important. 

5.6 Impacts of CEP on LEA Revenues from the School Meal Programs 

A key question about the CEP is whether it increases Federal reimbursements to LEAs. Section 5.6.1 
presents results on the impact of the CEP on the Federal reimbursement per meal and total Federal 
revenue per student for LEAs participating in the CEP, compared with LEAs not participating in the 
CEP. 

The impact of the CEP on revenues from non-Federal Sources is of particular interest to State and 
local decision-makers. States anticipating the national rollout of the CEP will want to know whether 
the implementation of the provision is associated with an increase in State funding for LEA 
foodservice operations. Similarly, stakeholders at all levels will want to know whether implementing 
the CEP leads to an increase in local funding for foodservice programs. Section 5.6.2 addresses these 
questions, explores the relationship between implementation of the CEP and revenues from payments 
for non-reimbursable foods, and considers the potential relationship between the provision and 
revenues from student payments for reimbursable meals. 

5.6.1 Federal Revenues 

As was the case for participation, the evaluation considered Federal reimbursement outcomes for the 
NSLP and SBP, using the Federal reimbursement rate (i.e., average dollars per meal) for the NSLP 
and SBP. In addition, the analysis considered impacts on total Federal revenue per student. These 
analyses used the LEA-level CITS analysis described in Section 5.4 (additional details and results for 
the full models appear in Appendix 5A). 

Federal Reimbursement per Meal for NSLP and SBP 

The impact of the CEP on revenue was examined first by comparing the Federal reimbursement per 
reimbursable meal for NSLP before and after the intervention among those LEAs that took up the 
CEP relative to a matched comparison group of non-participating LEAs. Exhibit 5.8 displays the 


83 This statistic is for all participating LEAs, not just those in the analysis sample. 
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regression-adjusted means and pre-post differences for the treatment LEAs and comparison LEAs and 
the estimated impact of the CEP. (See the table for important notes.) The average Federal 
reimbursement per meal for NSLP for the treatment LEAs increased $0.30 (from $2.45 to $2.75) 
compared to an increase in the matched comparison LEAs of $0.18 (from $2.31 to $2.49) over the 
same time period. The net difference between the increases observed in the treatment versus 
comparison LEAs is equal to a $0.13 larger increase in the treatment LEAs. This estimated impact of 
the CEP is statistically significant at the 1 percent level. In relative terms (the impact estimate divided 
by the comparison group’s pre-intervention mean), the CEP’s effect was to increase the Federal 
reimbursement per meal by 5.6 percent. 


Exhibit 5.8: Impact on Federal Reimbursement per Meal for NSLP 






Percent 


Comparison 

Treatment 

Impact 

Impact 

Mean Pre-Intervention Federal 
Reimbursement per Meal 

2.31 

2.45 



Mean Post-Intervention Federal 
Reimbursement per Meal 

2.49 

2.75 



Difference = (Post-Pre 

Intervention Federal 
Reimbursement per Meal) 

0.18 

0.30 

0.13*** 

5.6% 

Number of LEAs in sample 

525 

286 




For the purposes of this table, the average treatment effects are for Year 1 States LEAs who adopted the 
intervention in SY 2011-12. The mean pre-intervention Federal reimbursement per meal is the average of the 
regression- adjusted Federal reimbursement per meal for 2009 and 2010. The mean post-intervention Federal 
reimbursement per meal is the average of the regression-adjusted Federal reimbursement per meal for 2011 
and 2012. The pre- and post-intervention time periods for LEAs that implemented the CEP in 2012 are different, 
and so the regression-adjusted means are different, but the estimated impact is the same because it derives 
from the same parameter in the model. 

***=p<0.01, **=p<0.05, *=p<0.10. 

Source: State-provided administrative data. 

The same method was used to estimate the CEP’s impact on Federal reimbursement per meal for the 
SBP. As shown in Exhibit 5.9, 84 the average Federal reimbursement per meal for SBP for the 
treatment LEAs increased $0.11 (from $1.67 to $1.78 ) while the increase in matched comparison 
LEAs’ Federal reimbursement per meal was $0.08 (from $1.64 to $1.72 ) over the same time period. 
Thus, the increase in CEP-participating LEAs exceeded the increase in non-participating LEAs by a 
net $0.03. This impact of the CEP is statistically significant at the 1 percent level, and represents a 2 
percent increase in the Federal reimbursement per meal. 


84 As in Exhibit 5.1, estimates are for LEAs that implemented the CEP in SY 2011-12. 
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Exhibit 5.9: Impact on Federal Reimbursement per Meal for SBP 


Comparison 

Treatment 

Impact 

Percent 

Impact 

Mean Pre-Intervention Federal 
Reimbursement per Meal 

1.64 

1.67 



Mean Post-Intervention Federal 
Reimbursement per Meal 

1.72 

1.78 



Difference = (Post-Pre 

Intervention Federal 
Reimbursement per Meal) 

0.08 

0.11 

0.03*** 

1.9% 

Number of LEAs 

507 

286 




For the purposes of this table, the average treatment effects are for Year 1 States LEAs who adopted the 
intervention in SY 2011-12. The mean pre-intervention Federal reimbursement per meal is the average of the 
regression-adjusted Federal reimbursement per meal for 2009 and 2010. The mean post-intervention Federal 
reimbursement per meal is the average of the regression-adjusted Federal reimbursement per meal for 2011 
and 2012. The pre- and post-intervention time periods for LEAs that implemented the CEP in 2012 are different, 
and so the regression-adjusted means are different, but the estimated impact is the same because it derives 
from the same parameter in the model. 

***=p<0.01, **=p<0.05, *=p<0.10. 

Source: State-provided administrative data. 

Total Federal Revenue per Student 

The same method was used to estimate the CEP’s impact on total monthly Federal revenue from 
reimbursable meals per student enrolled, as an indicator of the overall level of Federal funding for 
school foodservice programs. Exhibit 5.10 displays the regression-adjusted means for the 
participating (treatment) FEAs and non-participating (comparison) FEAs. 85 The average total Federal 
revenue per student for the treatment FEAs increased $9.51, while the matched comparison FEAs’ 
total Federal revenue per student increased $4.18. Thus, the estimated impact of the CEP on total 
Federal revenue per student was $5.33, which is statistically significant and represents about a 13.5 
percent increase in the total Federal revenue per student. 


Exhibit 5.10: Impact on Total Federal Revenue per Student Enrolled 






Percent 


Comparison 

Treatment 

Impact 

Impact 

Average Pre-Intervention Total 
Federal Rev per Student 

$39.48 

$44.50 



Average Post-Intervention Total 
Federal Rev per Student 

43.65 

54.01 



Difference = (Post-Intervention 
Total Federal Rev per Student - 
Pre-Intervention Total Federal 

Rev per Student) 

4.18 

9.51 

$5.33*** 

13.5% 

Number of LEAs 

525 

286 



Number of observations 

2,025 

1,120 




***=p<.01, **=p<.05, *=p<.10. 

Source: State-provided administrative data. 


85 As in Exhibit 5.1, estimates are for LEAs that implemented the CEP in SY 2011-12. 
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5.6.2 Non-Federal Revenue 

The analysis of non-Federal revenue sources considered the impact of the CEP on revenue from State 
payments, transfers from other local sources to foodservice, payments for non-reimbursable foods and 
student payments for reimbursable foods. These analyses used an approach similar to that used to 
estimate the impact of the CEP on student participation rates and revenues from Federal sources. The 
analyses used data on revenue from four school years (2009-10, 2010-11, 2011-12, and 2012-13) 
collected primarily through the PEAR survey. 86 For LEAs in Year 1 States (Illinois, Kentucky and 
Michigan), this time period includes two pre-intervention and two post-intervention school years. For 
LEAs in Year 2 States (New York, Ohio and West Virginia), the time period includes three pre¬ 
intervention and one post-intervention school year. 87 Although multiple years of pre-intervention data 
were available, preliminary analyses of the pre-intervention years did not demonstrate a pattern of 
increasing or decreasing revenue for any of the non-Federal revenue types. For this reason, the 
variable representing time was omitted from the CITS model. 

Fall State Revenue per Student 

To estimate the CEP’s impact on State revenue, pre- and post-implementation measures of fall State 
revenue per student for participating LEAs were compared to those from non-participating LEAs. 88 
Exhibit 5.11 displays the regression-adjusted means for the participating (treatment) LEAs and non¬ 
participating (comparison) LEAs. 89 The total fall State revenue per student for the treatment LEAs 
increased $1.38 whereas the matched comparison LEAs’ fall State revenue per student decreased by 
$1.11 over the same time. Thus, compared to what would have happened in the absence of CEP 
participation, treatment LEAs experienced a net increase of $2.49 per student in State revenues. 
However, this estimated impact of CEP is not statistically significant, so one cannot conclude that the 
CEP increased State revenues per student. 


Exhibit 5.11: Impact on State Revenue 



Comparison 
($ per student) 

Treatment 
($ per student) 

Impact 

Mean Pre-Intervention State Revenue 

4.33 

12.89 


Mean Post-Intervention State Revenue 

3.23 

14.27 


Difference = (Post-Pre Intervention State 
Revenue) 

-1.11 

1.38 

2.49 

Number of LEAs in sample 

45 

54 



***=p<.01, **=p<.05, *=p<.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey; State-provided administrative data. 


86 The analysis of State Revenue combined data from State Administrative Data (Illinois PEAR survey 
respondents) and from the PEAR survey (PEAR survey respondents in Kentucky, Michigan, New York, 
Ohio and West Virginia). The analyses of Local Revenue, Payments for Non-Reimbursable Foods and 
Student Payments for Reimbursable Foods only used PEAR survey data. 

87 Data on the 2009-10 school year were not collected for New York LEAs. 

88 State revenue per student was used as the outcome because State revenues were not broken down between 
NSLP and SBP. 

89 As in Exhibit 5.1, estimates are for LEAs that implemented the CEP in SY 2011-12. 
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As discussed in Section 5.4, the matching process used to select a comparison group for this analysis 
was not fully successful in balancing all of the pre-intervention characteristics with those in the 
treatment group (i.e., the participating LEAs). As a result, the model used to estimate the impact of 
CEP on State revenues included covariates for the unbalanced baseline characteristics: ISP, percent of 
Title I schools, charter status and the percent of students who are Hispanic. 90 

Local Revenue 

A similar approach was used to estimate the CEP’s impact on the proportion of LEAs that transferred 
funds from other local sources to the foodservice program. Exhibit 5.12 displays the regression- 
adjusted mean proportion of treatment and comparison LEAs that received funding from local 
sources. 91 The proportion of treatment LEAs that received local funds decreased 3.7 percentage 
points, compared to a decrease of 1.2 percentage points in the comparison LEAs during the same 
time. The estimated impact of the CEP on the proportion of LEAs is a non-statistically-significant 
decrease of 2.5 percentage points. Thus, the analysis did not detect any change in the proportion of 
LEAs that transferred funds from other local sources to the foodservice program that resulted from 
CEP participation. 


Exhibit 5.12: Impact on Presence of Local Revenue 



Comparison 

Treatment 

Impact 

Proportion of LEAs Reporting Local Revenue 
Pre-Intervention 

26.5 

24.7 


Proportion of LEAs Reporting Local Revenue 
Post-Intervention 

25.3 

21.0 


Difference = (Post-Intervention - Pre- 
Intervention) 

-1.2 

-3.7 

-2.5 

Number of LEAs in sample 

64 

97 


Number of observations 

283 

389 



***=p<.01, **=p<.05, *=p<.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey; State-provided administrative data. 


Revenue from Payments for Non-Reimbursable Foods 

The impact of the CEP on the proportion of LEAs that generate revenue from payments for non¬ 
reimbursable foods is estimated using the approach described in for local revenue. As shown in 
Exhibit 5.13, the proportion of treatment LEAs that generated revenues from non-reimbursable foods 
decreased 6.2 percentage points compared to a decrease of 2.8 percentage points in the proportion of 
comparison LEAs generating such revenue. The difference between these differences is the impact of 
the CEP on the proportion of LEAs generating revenues from the non-reimbursable foods: a decrease 
of 3.5 percentage points, which is not statistically significant. However, an alternative specification of 
the impact model (used to check the robustness of the principal analysis result) found a statistically 
significant result (as discussed in Appendix 5A.) Thus, there is some suggestive evidence that the 
CEP reduces the probability that an LEA will generate revenue through sales of non-reimbursable 
foods, although the analyses did not produce a firm conclusion. 


90 The full results of this analysis are available in Appendix 5A. 

91 These results are based on a linear probability model following the specification in section 5.4 above. A 
longitudinal logistic regression confirmed the findings of the linear probability model. 
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Exhibit 5.13: Impact on Presence of Non-Reimbursable Foods Revenue 


Comparison 

Treatment 

Impact 

Proportion of LEAs Reporting Non- 
Reimbursable Foods Revenue Pre-Intervention 

66.6 

70.1 


Proportion of LEAs Reporting Non- 
Reimbursable Foods Revenue Post-Intervention 

63.9 

63.9 


Difference = (Post-Intervention - Pre- 
Intervention) 

-2.8 

-6.2 

-3.5 

Number of LEAs in sample 

61 

86 


Number of observations 

232 

325 



***=p<.01, **=p<.05, *=p<.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey; State-provided administrative data. 


Revenue from Student Payments for Reimbursable Meals 

Exhibit 5.14 presents the regression-adjusted mean revenue from student payments for reimbursable 
meals. Among treatment LEAs the regression-adjusted mean student revenue decreased by $12.83 per 
student, while the mean student revenue for comparison LEAs decreased by $4.58 per student. This 
suggests that the implementation of the CEP corresponds to a reduction in student revenues of $8.24 
per student. 


Exhibit 5.14: Impact on Revenue from Student Payments for Reimbursable Meals 



Comparison 
($ per student) 

Treatment 
($ per student) 

Impact 

Mean Pre-Intervention Student Payments 

69.13 

79.76 


Average Post-Intervention Student Payments 

64.55 

66.93 


Difference = (Post-Pre-Intervention Student 
Payments) 

-4.58 

-12.83 

-8.24 

Number of LEAs in sample 

29 

53 



***=p<.01, **=p<.05, *=p<.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey; State-provided administrative data. 


The regression-adjusted means presented in Exhibit 5.14 have already accounted for the baseline 
differences explained by the baseline covariates included in the model: ISP and percent of Title I 
schools. 92 These covariates were selected based on the observed imbalances in the sample at baseline. 

The magnitude of this estimate is biased downwards, potentially causing the analysis to miss a true, 
negative impact. The bias results from the construction of this outcome measure, which 
systematically over-represents LEAs that report generating no revenue from student payments for 
reimbursable meals as mentioned above and detailed in Appendix 5A. Hence, the student payments 
outcome measure underestimates student payments, on average. The magnitude of the underestimate 
appears to differ systematically between comparison and treatment LEAs at baseline. 93 Further, the 
magnitude of the underestimate is expected to be directly affected by the implementation of the CEP, 
which requires that fully participating LEAs generate no revenue from student payments. Had 
administrative data on student revenue been available, the findings of the impact of the CEP on 


92 The full results of this analysis are available in Appendix 5A, Exhibit 5A.26. 

93 See Exhibit 5A.26 in Appendix 5A for details. 
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student payments for reimbursable meals would likely have differed from those detailed above, given 
that participating schools do not collect payment from students for reimbursable meals. 

5.6.3 Discussion of Revenue Results 

The CEP’s impact on Federal reimbursements per meal were positive and significant for both 
outcomes (about 6 percent for the NSLP and 2 percent for the SBP) and over various alternative 
specifications (see Appendix 5A). The same was true of total Federal revenue per student, which had 
a 13.5 percent increase. No impact of the CEP was found on any of the types of non-Federal revenue. 
In all cases, these impact estimates must be interpreted carefully, as was the case for participation 
rates. These estimates do not generalize to FEAs in these States that are not comparable to the 
participating FEAs, nor do they generalize to FEAs in other States. 

As described in the discussion of participation results, the analyses depend on how well the increases 
in revenues in untreated comparison group represent what would have happened to schools in the 
treatment group had they not chosen to participate in the CEP. Because LEAs chose which schools 
adopted the CEP, participating and non-participating schools may differ systematically in ways that 
directly affect the outcomes of interest. However, this problem is likely modest, given that in the 
average participating LEA, 94 percent of eligible schools participated. 94 While the propensity score 
matching approach identified comparison LEAs with similar characteristics to their treatment 
counterparts, these characteristics do not account for other, potentially important, and unmeasurable 
characteristics that may influence the take-up decisions of LEAs, such as the expectation that the 
LEA will benefit from adopting the CEP. Nevertheless, the propensity score matching and the 
difference-in-difference analytic approach provide the strongest possible way to separate the impact 
of the CEP from the other factors that affected NSLP and SBP participation in the sample. Moreover, 
the robustness of the results provides additional confidence in the conclusion that the CEP did indeed 
increase Federal reimbursement per meal for NSLP and SBP and the total Federal revenue per 
student. The estimated relative impacts (5.6 percent Federal reimbursement per NSLP lunch and 13.5 
percent for total Federal revenue per student) are of sufficient magnitude that they are likely to be 
seen as not only statistically significant but also substantively important. However, the estimated 
impact on Federal reimbursement per meal is quite small relative to the cost of a paid meal, so LEAs 
would have to have a very low percentage of paid meals to have the same total revenue per meal. 

This finding is relevant to considerations of whether the current multiplier for claiming percentages is 
equitable. Taken together, the finding of increased Federal funding and the lack of evidence that 
other revenues significantly declined suggest that the CEP did not have an adverse overall effect on 
LEA foodservice revenues, and may have produced a net gain for participating LEAs. 

5.7 Impact on the Availability of the School Breakfast Program and Types of 
School Breakfast Service 

This section draws on a variety of data sources, including the Participation, Enrollment, Attendance 
and Revenue (PEAR) Survey, the Implementation Web Survey of Participating LEAs and the Pre¬ 
visit Questionnaire administered as part of the Menu Survey. 95 As each of these data sources draws 


Q4 This statistic is for all participating LEAs, not just those in the analysis sample. 

95 See Chapter 2 for more information on each data source and the sample for which data are available. 
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on slightly different sample, key characteristics of the sample and data source are mentioned in the 
discussion of the results for each question below. 

5.7.1 Presence of the School Breakfast Program 

Although the CEP requires that all participating schools offer the School Breakfast Program (SBP), 
the potential impact of the CEP on the presence of the SBP is limited to schools that do not offer the 
SBP and are among those schools that are likely to implement the CEP. 

Exhibit 5.15 below displays the unadjusted proportions of LEA PEAR survey respondents that 
offered the SBP in SY 2012-13 at all schools, by treatment group and State. Among PEAR 
respondents in the comparison group, 94 percent offered the SBP at all schools; among PEAR 
respondents in the treatment group of LEAs, 99 percent offered the SBP at all schools. Many States 
mandate that the SBP be offered at schools where more than a certain fraction of students qualify for 
free or reduced price lunch (FRAC, 2013). Among the 252 LEAs in the PEAR survey sample, only 8 
LEAs did not offer the SBP at all schools (6 treatment and 2 comparison). 96 

Exhibit 5.15: Proportion of LEAs that Offer the SBP at All Schools, by Treatment Group and 
State 




Comparison 


Treatment 


N 

SBP in All Schools 
(Percent) 

N 

SBP in All Schools 
(Percent) 

Illinois 

19 

94.7 

23 

100.0 

Kentucky 

24 

95.8 

24 

100.0 

Michigan 

32 

90.6 

43 

95.3 

New York 

8 

87.5 

12 

100.0 

Ohio 

10 

100.0 

26 

100.0 

West Virginia 

15 

100.0 

16 

100.0 

Total 

108 

94.4 

144 

98.6 


Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey. 


Exhibit 5.16 below presents the regression-adjusted proportion of LEAs that offer SBP at all schools. 
As the sample is unbalanced on ISP and average daily participation (ADP) in SBP, these means are 
obtained from a regression that controlled for ISP and SBP ADP. 97 The analysis found no effect of 
the CEP on the presence of the SBP. 


9(1 The two treatment LEAs that did not offer the SBP at all schools have not implemented the CEP district 
wide. The requirement to offer the SBP at all CEP schools does not apply at non-CEP schools in partially 
participating LEAs. 

1 These results were obtained from an ordinary least squares regression assuming a linear probability model. 
A logistic regression yielded the same results: the difference is not significantly different from zero when 
controlling for the covariates on which the sample was imbalanced. Baseline balance tests for this sample 
appear in Exhibit 2.7. 
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Exhibit 5.16: Impact on Presence of SBP 


Comparison 

Treatment 

Difference 


(Percent) 

(Percent) 

(Percentage Points) 

Regression-adjusted proportion 
of LEAs with SBP at all schools 

90.4 

94.1 

3.7 

Number of LEAs 

103 

142 



***=p<.01, **=p<.05, *=p<.10. 

Source: Participation, Enrollment, Attendance, and Revenue (PEAR) survey. 


The absence of observed impact on whether LEAs offer SBP does not preclude the possibility that 
CEP participation impacted the availability of the School Breakfast Program in other ways. Other 
data sources suggest such impacts. For example, among the 347 participating LEAs that responded to 
the Implementation Web Survey, 9 percent reported implementing or expanding their breakfast 
program due to the CEP. Below, results of correlational analyses of the relationship between 
participation and changes in SBP are reported, beginning with the relationship between participation 
in the CEP and the type of breakfast service offered. 

5.7.2 Type of Breakfast Service Offered 

Through the CEP, schools offer free breakfast to all enrolled students. Because they do not have to 
collect payment for reimbursable meals, schools implementing the CEP have more flexibility in how 
they serve breakfast to their students. Thus, participating schools may differ from non-participating 
schools in terms of the types of breakfast service offered. This section examines the relationship 
between CEP participation and types of breakfast service. 98 Data are from the Pre-Visit Questionnaire 
administered to the sample of treatment and comparison schools selected for the Menu Survey (a sub¬ 
sample of the PEAR sample). 

Exhibit 5.17 presents the combination of types of breakfast service utilized in each sampled cafeteria, 
by treatment group. Foodservice directors indicated if their cafeteria used traditional line service, 
grab-and-go breakfast, in-classroom breakfast or other type of service. About one quarter of 
foodservice directors (24 percent) indicated multiple types of breakfast service. Traditional line 
service was the most co mm on mode of breakfast service in both groups: 94 percent of comparison 
and 74 percent of treatment cafeterias used traditional line service (either alone or in combination 
with other models) at breakfast. 99 In cafeterias utilizing traditional line service, another type of 
breakfast service was sometimes indicated as well: 18 percent of all comparison and 20 percent of all 


98 This section presents data on the characteristics of the School Breakfast Program from the Pre-Visit 
Questionnaire administered as part of the Menu Survey. In the recruiting process for on-site data collection, 
schools operating universal free meals programs other than the CEP were screened out. The survey was 
fielded to each cafeteria representing the schools selected for Component 3 on-site data collection. This 
analysis examines the experience of 146 cafeterias representing 156 schools. All cafeterias in both the 
treatment and comparison groups offer breakfast. 

99 Note that the proportion of cafeterias that used traditional line service is found by summing across all the 
rows in Exhibit 5.17 that include traditional line service. 
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treatment cafeterias reported both traditional line service and grab and go breakfast. Foodservice 
directors for four cafeterias indicated a type of service other than the listed types. 100 


Exhibit 5.17: Prevalence of Combinations of Types of Breakfast Service, by Treatment Group 


Type of Breakfast Service 

Comparison 

(Percent) 

Treatment 

(Percent) 

Traditional line service only 

75.0 

45.9 

Traditional and grab and go 

18.1 

20.3 

Grab-and-go breakfast only 

5.6 

9.5 

In-classroom breakfast only 

0.0 

13.5 

Traditional and in-classroom 

1.4 

2.7 

Traditional and other type of service 

0.0 

2.7 

Traditional, in-classroom and grab and Go 

0.0 

2.7 

Other type of service 

0.0 

1.4 

Grab-and-go and other type of service 

0.0 

1.4 

Number of cafeterias 

72 

74 


Source: School Pre-Visit Questionnaire/Menu Survey. 


Exhibit 5.18 compares the prevalence of each type of breakfast service between the comparison and 
treatment groups. 101 CEP participation is associated with a statistically significant reduction in the 
proportion of LEAs that utilize traditional line service at breakfast, and a statistically significant 
increase in the proportion of LEAs that utilize in-classroom breakfast. Although the proportion of 
LEAs utilizing grab and go breakfast is higher in the treatment group, the difference is not significant. 


Exhibit 5.18: Prevalence of Each Type of Breakfast Service, by Treatment Group 



Comparison 

Treatment 

Difference 

Traditional Line Service 

94.4% 

74.3% 

-20.1%*“ 

In-classroom 

1.4% 

18.9% 

17.5%“* 

Grab-and-go 

23.6% 

33.8% 

10.2% 

Number of LEAs in sample 

25 

27 


Number of cafeterias 

72 

74 



Percentages do not sum to 100 because schools could offer more than one of these models. 
***=p<.01, **=p<.05, *=p<.10. 

Source: School Pre-Visit Questionnaire/Menu Survey. 


To investigate which alternative types of breakfast service were retained as ongoing practice, rather 
than being implemented for a single year as one-time “experiment,” Exhibit 5.19 presents the 
prevalence of each model of breakfast service by duration of CEP participation in the Year 2 States 
(Illinois, Kentucky and Michigan). In these States, the treatment group included schools in their first 


100 The four foodservice directors that indicated another type of service described these services as “Food 
Court,” “Vending Machines,” “To Go Boxes/Student Choice” and “Cart Out to Common Area.” All of 
these cafeterias are in the treatment group. 

101 Each of these results were obtained from a regression that assumed a linear probability model, clustered 
standard errors by LEA and included no covariates. Logistic regression yielded qualitatively similar results. 
Although schools could offer more than one of each type of breakfast service, the analyses were conducted 
separately. As this analysis is exploratory, no adjustments were made to account for multiple comparison 
issues or correlations across outcomes. 
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year of implementation of the CEP as well as schools that in their second year of implementation. 
Among cafeterias that were in their second year of CEP, 71 percent utilized traditional line service, 
the same proportion as in cafeterias in their first year of implementation. In comparison group LEAs, 
97 percent of cafeterias utilized traditional line service. This pattern reinforces the finding presented 
in 5.19 that participation in the CEP is associated with a decrease in the use of traditional line service 
at breakfast. Similarly, the prevalence of in-classroom breakfast in comparison, first year treatment 
and second year treatment LEAs (0, 24 and 19 percent, respectively) reinforces the association 
between CEP participation and an increased likelihood of offering breakfast in the classroom. These 
patterns suggest that the changes in breakfast service are not the result of a one year trial of a new 
model of breakfast service, but an ongoing shift in practice. 


Exhibit 5.19: Prevalence of Models of Breakfast Service in Year 2 States (Illinois, Kentucky and 
Michigan), by Duration of CEP Participation 



Comparison 

First Year of 
Treatment 

Second Year of 
Treatment 

Traditional Line Service 

97.2% 

70.6% 

71.4% 

In-classroom 

0.0% 

23.5% 

19.0% 

Grab-and-go 

13.9% 

41.2% 

19.0% 

Number of cafeterias 

36 

17 

21 


Source: School Pre-Visit Questionnaire/Menu Survey. 


5.7.3 Student Choice of Foods at Breakfast 

To address the research question on whether the identical breakfast is served to all students, this 
section relies on data describing whether students have a choice of foods to select from at 
breakfast. 102 Exhibit 5.20 compares proportions of cafeterias in which students have a choice of 
breakfast between the comparison and treatment groups, after controlling for differences between the 
treatment and comparison schools. 103 While students have a choice of breakfast in 89 percent of 
treatment cafeterias, participation in the CEP is associated with a statistically significant, 9.4 percent 
smaller likelihood that a cafeteria will offer students a choice of food at breakfast. 


Exhibit 5.20: Percentage of LEAs reporting that Students Have a Choice of Foods at Breakfast, 
by Treatment Group 



Comparison 

Treatment 

Difference 

Students have a choice of foods 
at breakfast 

98.6% 

89.2% 

-9.4%“ 

Number of LEAs in sample 

25 

27 


Number of cafeterias 

72 

74 



Source: School Pre-Visit Questionnaire/Menu Survey. 


102 As in the previous section, this section presents data on the characteristics of the School Breakfast Program 
from the Pre-Visit Questionnaire administered as part of the Menu Survey. The survey was fielded to each 
cafeteria representing the schools selected for Component 3 on-site data collection. This analysis examines 
the experience of 146 cafeterias representing 156 schools. All cafeterias in both the treatment and 
comparison groups offer breakfast. 

103 These results were obtained from a regression that assumed a linear probability model, clustered standard 
errors by LEA and included no covariates. Logistic regression yielded qualitatively similar results. 
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5.7.4 Time Allowed for Students to Eat Breakfast 

In both the treatment and comparison groups, the typical cafeteria allows students 30 minutes to eat 
breakfast. 104 These data are obtained from the Pre-Visit Questionnaire administered to Component 3 
LEAs as part of the Menu Survey. 


Exhibit 5.21 displays the distribution of the length of the breakfast period in minutes for the treatment 
group, the comparison group, and all cafeterias. The distribution is very similar across the three 
groups. 


Exhibit 5.21: Length of Breakfast Period, by Treatment Group 



Length of Total Breakfast Period (minutes) 


Comparison 

■ Treatment 

■ Total 


Source: School Pre-Visit Questionnaire/Menu Survey. 

5.7.5 Discussion of School Breakfast Program Results 

The School Breakfast Program was widely available in LEAs that are not participating in the 
Community Eligibility Provision but are likely to take up the CEP: 105 among LEAs in the matched 
comparison group that completed the PEAR survey, 94 percent offered the SBP at all schools in 
school year 2012-13. This level of availability of the SBP in LEAs likely to implement the CEP 
means that relatively few LEAs implementing the CEP option will be required to offer a “new” 
breakfast program. Indeed, the data from the PEAR survey demonstrated no statistically significant 
difference in SBP availability between the treatment and comparison groups. This finding suggests 
that the requirement that schools participating in CEP offer a SBP is not a barrier to implementation 
of the CEP, but also that the CEP will not increase the availability of the SBP. 


104 The median and modal length of breakfast period is 30 minutes for both groups. The mean is slightly higher 
because a few cafeterias reported breakfast periods of 75 minutes. 

105 Comparison LEAs were selected through a propensity score matching process that identified them as 
approximately equally likely to take up the CEP as a treatment LEA. See Chapter 2 for details. 
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Other findings demonstrate that implementation of the Community Eligibility Provision is associated 
with changes in the model of breakfast service. Implementation of the CEP is associated with a 
decrease in the use of traditional line service at breakfast and an increase in the use of in-classroom 
breakfast, which is often recommended as a way to increase SBP participation but complicated to 
implement under conventional meal counting and claiming procedures. 

Participation in the CEP is also associated with a decrease in the likelihood that students are offered a 
choice of foods at breakfast. The reduction in choice of foods is potentially linked to the change in the 
model of breakfast service. A shift from traditional line service to in-classroom breakfast could 
plausibly involve a move from service that offers students choice to one that does not. 

Although the sample used to investigate the impact on ADP in the SBP is quite different and much 
larger than the one used to investigate the characteristics of the SBP, taken together, the two analyses 
tell a coherent story. Both in-classroom breakfast and reduced choice represent strategies that an LEA 
might use to increase the number of breakfasts served within the constraints of its schedules and 
facilities. This interpretation is consistent with the increase in average daily participation in the 
School Breakfast Program described earlier in the chapter. 
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6. CEP Impacts on Staffing, Administrative, and Meal Production 
Costs 


This chapter reports the effects of CEP participation on staffing, administrative, and meal production 
costs, specifically addressing three research questions: 

1. To what extent did the administrative burden of operating the NSLP and SBP change under the 
CEP? 

2. What is the difference in administrative costs associated with the Option to the LEA compared to 
the costs in the absence of the option? 

3. To what extent did the costs of producing reimbursable meals change under the CEP? 

All data on administrative cost and staffing impacts were collected from samples of treatment (CEP) 
and comparison (non-CEP) LEAs and schools. The comparison schools were selected from LEAs 
matched to the treatment LEAs using the propensity score method described in Chapter 2. The 
estimates of impacts on administrative costs and staffing are specific to the respective samples of 
schools observed. These estimates do not represent the impact of CEP on other participating schools 
not represented by the sample, including schools that did not meet the sampling criteria and schools in 
other States. 

All impact measures are based on a posttest-only comparison group design, which relies on the 
matched sample and regression models that control for differences between treatment and comparison 
schools other than whether they use the CEP. It is important to note that analyses based on data 
collected after a policy has been implemented may be subject to selection bias or may reflect 
confounds arising from other systematic differences. For example, because the evaluation has no pre¬ 
program observations for either the treatment or comparison group members, one cannot kn ow 
whether the posttest observations differ from the pre-intervention levels. As discussed in Chapter 2, 
the use of matched comparison groups and regression models reduces, but does not eliminate, the 
selection bias inherent in comparisons when treatment is voluntary. In particular, the posttest-only 
comparison design does not control for unobserved differences that are not highly correlated with the 
known characteristics of the LEAs and schools in the sample. Nevertheless, this was the most feasible 
strategy for estimating the impact of the CEP, by providing some contrast to a non-CEP condition 
that is the best estimate of the counterfactual that can be identified. As such, the method represents an 
improvement over a purely descriptive estimate of treatment group outcomes, by accounting for 
systematic differences in school characteristics that may affect the outcomes. 

The analyses in Chapters 6, 7, and 8 should be considered exploratory and viewed with more caution 
than the analyses in Chapter 5. As discussed in Chapter 2, the evaluation included a large number of 
outcomes, and so the design specified two outcomes as confirmatory (NSLP participation and Federal 
NSLP revenue per meal) and the rest of the outcomes as exploratory. The statistical tests have not 
been adjusted for multiple comparisons, i.e., the likelihood of finding a statistically significant impact 
in numerous tests as a result of chance. Therefore, readers are advised to view individual results as 
only suggestive of probable impacts, unless they have an especially high degree of confidence (i.e., 
significance at the 1 percent level), and to focus on more systematic patterns of similar results for 
related outcomes. 
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6.1 Expectations for CEP Impacts on Staffing, Administrative, and Meal 
Production Costs 

A major objective of the CEP is to eliminate the time and cost burden of processing applications for 
FRP meals. CEP participation also has the potential to reduce effort and costs associated with 
counting meals, collecting student payments for meals, and claiming reimbursements. Moreover, CEP 
participation may alter the per meal production costs if schools simplify menus or otherwise 
streamline operations to meet increased demand associated with CEP participation, or if fixed costs of 
foodservice operations are spread over an increased number of meals served due to increased student 
participation. 

Although the CEP is expected to reduce the costs of SBP and NSLP administration overall, the extent 
of this reduction is uncertain. While participating LEAs and schools are not required to process and 
verify FRP applications, some LEAs may collect alternate household income forms for purposes 
other than those required by FNS. Although the costs of doing so should not be attributed to school 
meal programs and were not measured for this evaluation, from a broader perspective they remain 
part of the LEA’s overall administrative costs. Direct certification effort and costs could potentially 
increase with CEP participation, if LEAs and schools devote more effort to maximizing their ISPs 
either to qualify for the CEP or to achieve higher reimbursement rates. On the other hand, CEP LEAs 
are not required to conduct direct certification, so effort and costs for this function could drop. 
Similarly, LEAs and schools participating in the CEP might simplify meal counting and claiming 
procedures, yet some may maintain existing processes to assure accurate meal counts and avoid 
duplicate participation. Anticipated reductions in meal production costs may not be realized if LEAs 
use the cost savings from other tasks to upgrade menus, resulting in the same or higher meal 
production costs. It is also worth noting that costs associated with all of these activities could initially 
rise as LEAs and schools transition to new systems. 

6.2 Outcome Measures 

To address these issues, this chapter presents analytic impact estimates on administrative effort and 
costs in two ways. First, it presents impacts for the five functions that were expected to be affected by 
the CEP: 

A. Distributing and processing applications for free or reduced-price meals; 

B. Direct Certifications; 

C. Verifying income of free/reduced price students; 

D. Meal payment collections and accounting; and 

E. Counting and claiming reimbursable meals. 

CEP participation could also shift the distribution of effort and cost across different staff within the 
LEA. Thus, to address this question, this chapter also presents estimated impacts on administrative 
effort and costs for four staff types: 

A. LEA staff; 

B. School foodservice staff; 

C. School non-foodservice staff; and 

D. School staff, role indeterminate or mixed. 
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The final section of this chapter provides results on LEA foodservice directors’ impressions of cost 
impacts based on questions from the PEAR survey and the administrative cost and staffing interview. 

6.3 Data and Methods 

6.3.1 Data 

The impact estimates in this and the next section are derived using the posttest-only comparison 
group design outlined in Chapter 2. Chapter 2 also described the data collection for these analyses. 106 
Briefly, average hourly labor rates were calculated for each position in each school and in the LEA 
central office based on staff roster data. The hourly rates were then linked to data on annual hours 
spent by each staff position on the applicable administrative tasks to derive an annual cost. 107 The 
annual time and labor cost data were then divided by the school’s enrol lm ent (for school-level tasks) 
or LEA enrollment (for LEA-level tasks) and summed by task and position type for each school or for 
the LEA. The LEA-level time and labor costs per student were added to the corresponding time and 
labor costs for each school (this entails the assumption that LEA-level costs for each school are 
proportional to enrollment). The analytic variables represent each school’s per student annual hours 
and labor costs for each administrative task and position type. As noted in Chapter 2, the analysis file 
comprised data on 247 schools (123 treatment and 124 comparison) in 100 LEAs (52 treatment and 
48 comparison). Appendix 6A presents detail about the characteristics of the analysis sample. 

Five questions from the PEAR survey pertained to changes in meal production costs over the past two 
years. An average of 218 respondents (129 treatment and 89 comparison LEAs) replied to these 
questions. The results are tabulated in Section 6.5. At the end of the administrative cost interview, 
conducted as part of Component 2 data collection, nine questions were asked of CEP participating 
LEAs. There were 54 respondents to these interview questions (some LEAs had more than one 
respondent). Two of the nine questions pertained to meal production costs. The results are also 
presented in Section 6.5. 

6.3.2 Analytic Model 

The impact results presented in this chapter (and in Chapters 7 and 8) rely on the posttest-only 
comparison group design to assess the impact of the CEP administrative staff and time costs for tasks 
associated with FRP certification and meal counting and claiming. Schools were used as the unit of 
analysis, since a significant portion of these activities occur at the school level (as opposed to LEA 
central offices). The model used to estimate impacts was: 

Y si = Po + Pi T i + /? 2 Z + £ si (Eq. 6.1) 


where 


106 Complete details on the construction of the cost analysis file will be provided in the final database and 
accompanying documentation deliverable. 

107 The original plan was to include position-specific data on fringe benefits rates and LEA-specific data on 
indirect cost rates in the hourly rate calculations. However, inconsistencies in the availability of some data 
elements required the use of common fringe and indirect cost rates of 30 and 12.75 percent, respectively 
(based upon the study sample median rates) 
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Y s i is the annual labor time or cost per enrolled student for school s in LEA i; 

7) is the treatment indicator (=1 if LEA i participates the CEP; =0 otherwise); 

Z si is a vector of characteristics of school s in LEA i; and 
£ si is the error term. 

The error tenn is assumed to be correlated within LEAs, and the standard errors of coefficients are 
adjusted to account for this clustering. 

The coefficient of the treatment indicator, /? l5 is the impact estimate. The coefficient vector /? 2 reflects 
the effects of other variables included in the model as statistical controls. The first parameter, /? 0 , 
describes the hypothetical outcome in the absence of the treatment and with all the Z variables set to 
zero. 

The vector of school characteristics, Z, included: 

• the school’s grade level (i.e., whether elementary, middle, or high), with high school as the 
reference category; 

• the State, with West Virginia as the reference State; and 

• school enrollment, school percentage of students certified for FRP meals, and LEA ISP (the LEA 
and school-level features where balance testing discerned a difference between treatment and 
comparison LEAs that was statistically significant at the 10 percent level). 

The impact estimates presented in the results section draw on the results of the above model. The 
impact estimate, or the coefficient of the treatment indicator, /? ls is labeled “T-C Difference (Impact)” 
in these Exhibits. The column labeled “Comparison schools” is the average of the outcome for 
schools in the comparison group, and the column labeled “Treatment schools” is the average of the 
outcome for schools in the comparison group plus the impact estimate. This latter quantity may be 
thought of as a regression-adjusted average outcome for schools in the treatment group, or, 
alternatively, the average outcome schools in the comparison group if they participated in the CEP 
program. 

Additional details on the data used in the model, the balance tests used to determine which covariates 
to include, and the complete regression model results are presented in Appendix 6A. 

6.4 CEP Impacts on Administrative Staff Time and Costs 

6.4.1 CEP Impacts on Administrative Staff Effort and Costs Overall and by Type of Activity 

Estimated impacts of CEP participation on staff effort and costs devoted to the different types of 
administrative functions are shown in Exhibit 6.1. Results are displayed as regression-adjusted means 
for the treatment group, actual means for the comparison group, differences in means, and percentage 
differences in means. The CEP reduced staff time and costs for distributing and processing 
applications for free or reduced-price meals, verifying income of free/reduced price students, and 
meal payment collections and accounting, but increased staff time and costs for counting and 
claiming reimbursable meals. There was no impact on effort and time for direct certification, which 
was the function with the least time per student in the comparison group. 
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Exhibit 6.1: Estimated Impacts of CEP Participation on Annual Time and Labor Costs for 

Administrative Tasks, by Task Domain 






Comparison 

Treatment 

T-C Difference 

Percent 


Schools 

Schools 

(Impact) 

Impact 

Hours per enrolled student 

A. Distributing and processing 





applications for free or reduced-price 
meals 

0.46 

0.12 

-0.34*** 

-74.8 

B. Direct certifications 

0.09 

0.09 

0.00 

4.4 

C. Verifying income of free/reduced 
price students 

0.12 

0.01 

-0.11*** 

-94.2 

D. Meal payment collections and 
accounting 

1.92 

1.24 

-0.68“ 

-35.5 

E. Counting and claiming reimbursable 
meals 

0.43 

0.87 

0.44*** 

103.8 

Total 

3.01 

2.32 

-0.69 

-23.0 

Cost per enrolled student 

A. Distributing and processing 





applications for free or reduced-price 
meals 

13.78 

2.93 

-10.84*** 

-78.7 

B. Direct certifications 

3.45 

3.46 

0.01 

0.4 

C. Verifying income of free/reduced 
price students 

3.99 

0.11 

-3.88*** 

-97.1 

D. Meal payment collections and 
accounting 

41.71 

27.61 

-14.10* 

-33.8 

E. Counting and claiming reimbursable 
meals 

9.45 

21.54 

12.09*** 

127.9 

Total 

72.38 

55.66 

-16.72 

-23.1 

Number of schools 

124 

123 



Number of LEAs 

48 

52 




***=p<0.01; **=p<0.05; *=p<0.10. 

Source: Administrative Cost Interview and Staff Roster data. 


Total administrative time and costs for the five functions were lower for the treatment group but the 
differences were not statistically significant. 108 Schools in the comparison group averaged about 3 
hours and $72 per enrolled student per year on the specified administrative activities associated with 
the SBP and NSLP. The point estimates suggest that schools in the treatment group spent about 0.7 
hours (40 minutes) and $17 per student less than the comparison group. A modest reduction in overall 
administrative costs is consistent with data from the administrative cost interviews with foodservice 
directors in CEP LEAs, where 15 percent reported a reduction in administrative staffing and 6 percent 
reported re-assigning staff compared with 4 percent who reported increased administrative staffing. 

The largest impact of the CEP on labor time and costs occurred in two functions: distributing and 
processing applications for free or reduced-price meals, and verifying income of free/reduced price 
students. Comparison schools spent about a half hour, at a cost of $13.78, per student per year, 
distributing and processing applications for free or reduced-price meals; the CEP was estimated to 


108 The results for total administrative time and costs are not statistically significant, yet the pattern is 

suggestive, as results are significant under slight differences in model specification, and there are effects on 
the components of total costs, which, taken together, may point to plausible impacts on total costs. 


Abt Associates 


6. CEP Impacts on Staffing, Administrative, and Meal Production Costs | pg. 121 























Community Eligibility Provision Evaluation: Final Report 


reduce the time spent on this activity to 7 minutes and costs to less than $3 per student per year. 
Likewise, comparison schools spent nearly $4 per student annually verifying income of free/reduced 
price students; the CEP reduced this cost to $0.11. 109 

Most effort was spent on meal payment collections and accounting in both treatment and comparison 
groups. Consistent with expectations, CEP participation reduced the time and cost associated with this 
activity by about 35 percent. In terms of labor costs, for example, schools’ average annual costs per 
student associated with meal payment collections and accounting were $42 in comparison schools and 
$28 in treatment schools. 

Taken together, CEP participation appears to reduce time spent on distributing and processing 
applications for free or reduced-price meals, verifying income of free/reduced price students, and 
meal payment collections and accounting by 68 minutes, and reduces labor costs for these activities 
by about $29 per student per year. The CEP had no impact on the staff time and costs associated with 
Direct Certification. 

A surprising result is that the CEP more than doubled staff time and cost of counting and claiming 
reimbursable meals. Comparison schools devoted less than half an hour, or $9.45, per student per year 
to this activity; treatment schools spent almost an hour, or $21.54 per student per year. Increased 
participation in school meal programs—discussed in the preceding chapter—may partially explain the 
higher counting and claiming costs, but the difference seems too large to be fully accounted for by 
increased participation. 

States did not report any LEA concerns about increased time for meal counting and claims, and in 
fact expected that simplified procedures resulting from the CEP would result in less time needed for 
these processes. Qualitative data from LEA cost interviews suggest simplifications or cost savings— 
the opposite of analytic results—associated with meal counting processes. A substantial proportion 
of LEA foodservice directors (41 percent) reported that data processing, including systems of 
counting meals, had changed as a result of the CEP; of these FSDs, reported types of changes 
included the following: 

“Not using software; only counting how many students take breakfast/lunch daily.” 

“Saves money because they no longer need the POS. Simplified the entire process. POS systems 
are very expensive to buy, maintain, annual fees and support.” 

“Implemented tally sheet in elementary school—faster.” 

“At breakfast in elementary schools, students no longer have to enter ID number as they go through 
the line.” 


109 There are two possible explanations for why the estimated effort and cost for processing applications and 
verification are not zero for treatment schools. First, some treatment LEAs in the Component 2 sample had 
non-CEP schools, and these may have had LEA-level costs for application processing and/or verification. 
Second, although interviewers specifically asked about applications for FRP meals, some treatment schools 
or LEAs may have reported costs of processing and verifying household income forms collected in lieu of 
FRP meals applications. These costs are not true school meals program costs. Thus, some treatment schools 
or LEAs may have overstated their application processing and verification costs, so the actual reduction in 
effort and costs attributable to the CEP may be understated. 
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As discussed in Chapter 3, all six CEP States in the Impact Study provided automated systems for 
LEAs to submit claims for meal reimbursements. Some States modified their systems to automate the 
computation of free and paid meal counts, while others provided spreadsheets for LEAs to make these 
computations. In both cases, the LEAs had to use a new process, and this process may have taken 
longer than the conventional claims process, particularly when the LEA used a spreadsheet to 
compute the meal counts before entering them. 110 Also, some LEAs stopped using their point-of-sale 
systems and switched to manual procedures for counting meals, in order to speed up cafeteria lines 
(as noted in the quotes above). This change would contribute to the reduction in time for meal 
payments and collections, but it would then require workers to total up meal counts on the tally sheets 
or other manual documents. The CEP could produce net savings in meal counting and claiming costs 
in the long run, if schools no longer had to bear the cost of automated systems for meal counting and 
payments (provided that changes in cafeteria procedures do not increase labor costs by the same 
amount or more). Since data were not collected on these latter costs, such savings (if any) could not 
be reflected in the analysis. 111 

6.4.2 CEP Impacts on Administrative Staff Effort and Costs by Type of Staff 

The analytic results presented in this subsection are derived from the same data and methods 
described above. Here, however, impacts on different kinds of staff are examined. Although almost 
all of the point estimates for CEP impacts by staff type are negative, none are statistically significant. 

Exhibit 6.2 shows estimated CEP impacts on schools’ annual administrative staff time and labor costs 
per enrolled student for different kinds of staff. In comparison schools, school foodservice staff spent 
the most staff time (1.77 hours per student, or about 60 percent of the total staff time allocated to the 
specific administrative activities examined in this analysis), at a cost of $36 per student (50 percent of 
total labor costs analyzed). CEP participation was associated with roughly 20 percent less time and 
cost, although the difference was not statistically significant. 

LEA and school non-foodservice staff in comparison had roughly equivalent shares of the remainder 
of the effort and cost, amounting to about 30 minutes (one half-hour) or $14-19 per student per year 
for each. Except for school non-foodservice staff, the adjusted treatment group means are less for 
these staff types, implying a reduction in effort and costs, but none of the estimated impacts for any 
particular type of staff is statistically significant. These results, combined with those in the previous 
section, suggest that CEP changes what staff do while reducing the administrative effort of all staff. 

In other words, the time and cost savings on administrative tasks associated with CEP participation 
are not concentrated in a single category of worker, but are broadly dispersed. 


110 There was not a significant difference in meal counting and claiming costs between LEAs in the Year 1 and 
Year 2 States, or between LEAs with full versus partial CEP participation. However, the sample was not 
designed to test for such differences, or for differences between States with fully-automated claims systems 
and those that used a less automated process for the CEP. The interviews asked only about operational 
tasks performed by LEA staff, so the costs do not include modifications to LEA systems. 

111 Collection of data on automated systems costs was not attempted because such systems are usually 
integrated and support multiple functions, so the costs associated with the functions not needed under CEP 
would be difficult if not impossible to break out. Furthermore, systems to track meals served at the student 
level may be required to assure that the LEA claims only one meal per participating student. 
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Exhibit 6.2: Estimated Impacts of CEP Participation on Annual Time and Labor Costs for 
Administrative Tasks, by Staff Type 



***=p<0.01; **=p<0.05; *=p<0.10. 

Source: Administrative Cost Interview and Staff Roster data. 


6.5 Descriptive Findings for CEP Impacts on Meal Production Costs 


Impacts on meal production costs were assessed in two ways: the PEAR survey and interviews with 
CEP-participating LEA foodservice directors, both of which asked whether costs had increased or 
decreased and whether LEAs had allocated more or fewer staff to food production activities. 
Although responses to these questions cannot serve as hard evidence of cost impacts, they provide 
useful descriptive information. Overall, they present a mixed picture. Some results suggest cost 
savings or greater ease in breaking even, while others suggest increased costs. 

Exhibit 6.3 shows responses to the question asked of foodservice directors in CEP participating 
LEAs: “Have you made any changes to food production staffing (that is, the number of person hours) 
as a result of implementation of the Community Eligibility Option?” The majority of LEA (56 
percent) reported no change. Thirty-two percent reported an increase while 7 percent reported a 
decrease. This suggests that some CEP-participating LEAs may have increased total meal production 
effort in response to increased participation, but it does not necessarily indicate a change in 
production costs per meal. In fact, if the number of meals served increased more than staff costs, the 
staff cost per meal would go down. 

Exhibit 6.4 presents data about meal production costs from the PEAR survey administered to 
respondents in treatment and comparison LEAs. Large majorities (about 78 percent for breakfast and 
85 percent for lunch) of both treatment and comparison LEAs reported increased food costs over the 
past two years. Somewhat higher proportions of comparison than treatment LEAs reported higher 
labor costs (52 versus 47 percent for breakfast and a statistically significant 67 versus 55 percent for 
lunch). This suggests that meal production labor costs may have increased more for comparison than 
for treatment LEAs, and thus that the CEP may have helped to offset the labor cost increase as both 
groups implemented new meal pattern rules. However, slightly more treatment than comparison 
LEAs (56 versus 51 percent) indicated that cafeteria workers’ average hourly pay had increased over 
last two years, suggesting the opposite. 
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Exhibit 6.3: Responses to the question: “Have you made any changes to food production 
staffing (that is, the number of person hours) as a result of implementation of the Community 
Eligibility Option?” 

Percent 

Yes—reduced staff hours 

7.4 


Yes—increased staff hours 

31.5 


No changes 

55.6 


Don’t know 

1.9 


Missing 

3.7 


Total 

100.0 


Number of observations 

54 


Source: Administrative Cost Interview. 



Exhibit 6.4: Responses to PEAR Survey Meal Production Cost Questions 



Treatment 

Comparison 

Labor cost per breakfast increased over last two years 

47.3 

(N=129) 

51.8 

(N=85) 

Food cost per breakfast over the last two years 

78.5 

(N=130) 

78.4 

(N=88) 

Labor cost per lunch increased over last two years 

54.6 

(N=130) 

67.4 

(N=89) 

Food cost per lunch over the last two years 

85.0 

(N=133) 

85.4 

(N=96) 

Cafeteria workers’ average hourly pay increased over last two years 

55.6 

(N=124) 

51.1 

(N=88) 


Source: PEAR Survey. 


Finally, Exhibit 6.5 summarizes FSD responses (from CEP LEAs) about how CEP participation 
affected LEAs’ ability to balance revenues and costs. The results suggest that, for the majority of 
LEAs, the CEP has made it easier for LEAs to break even or at least had no impact. Forty-four 
percent of respondents replied that it was easier to break even under CEP and 19 percent reported no 
change, while 11 percent reported that it was more difficult. 


Exhibit 6.5: Responses to the question: “Has the Community Eligibility Option affected 
whether your LEA foodservice is able to break even, that is, whether revenues from all 
sources are at least equal to costs?” 


Percent 

Yes—easier 

44.4 

Yes—harder 

11.1 

No change 

18.5 

Don’t know 

18.5 

Did not respond to question 

7.4 

Total 

100.0 

Number of observations 

54 


Source: Administrative Cost Interview. 
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7. Program Integrity Impacts 


This chapter presents the findings about the impacts of the Community Eligibility Provision (CEP) on 
the integrity of the NSLP and SBP. The specific research questions regarding program integrity were: 

1. What is the impact on administrative errors associated with the certification process? 

2. What is the impact on errors in: 

a. meal counting by cashiers and 

b. computing claims for reimbursement? 

There are three dimensions of program integrity corresponding to these three questions. The CEP was 
expected to simplify program administration and therefore reduce errors in two of these dimensions: 
certification for FRP meals, and counting and claiming of reimbursable meals. For meal counting by 
cashiers (i.e., the identification of which meals are reimbursable), there were some reasons to expect 
reduced error, and some reasons to expect that errors could increase, as discussed below. 

All such errors are important to USDA, LEAs, and other stakeholders, because they may result in 
LEAs claiming too much or too little reimbursement. Certification errors result in too many or too 
few students being approved for FRP meals; this in turn affects the rate of reimbursement received by 
LEAs for meals taken by students (e.g., when ineligible students receive free meals, LEAs are paid 
more than when students were denied free meals and their meals were claimed at the paid meals rate). 
Meal counting errors by cashiers result in too many or too few meals claimed for reimbursement. 
Similarly, errors in computing claims for reimbursement also result in LEAs claiming too much or 
too little reimbursement. Improvements in program integrity can reduce improper payments to LEAs, 
increase legitimate payments to LEAs, or both. 

All data on program integrity impacts were collected in samples of treatment (CEP) schools and 
comparison (non-CEP) schools. The comparison schools were selected in LEAs matched to the LEAs 
in the treatment group using the propensity score method described in Chapter 2. Certification error 
data were collected in 248 schools in the Component 2 sample. Cashier error data were collected in 
155 schools in the Component 3 sample, and meal counting and claiming error data were collected in 
137 schools in the same sample. The estimates of impacts for each of these error types are specific to 
the respective samples of schools observed (for that type of data), including both selected treatment 
cases and their matched comparison counterparts. These estimates do not represent the impact of CEP 
on other participating schools not represented by the sample, such as schools that did not meet the 
sampling criteria and schools in other States. As discussed in Chapters 2 and 6, the comparisons may 
be subject to selection bias, to the extent that the matched sample and regression models do not 
control for all relevant differences between the treatment and comparison groups; however, the 
evaluation used the strongest feasible method for these analyses. Finally, because the analyses in this 
chapter are exploratory, readers are cautioned to consider individual results as only suggestive of 
probable impacts, unless they have an especially high degree of confidence (i.e., significance at the 1 
percent level), and to focus on more systematic patterns of similar results for related outcomes. 

This chapter provides separate discussions of the outcomes, data and methods, models, and results for 
each of the three dimensions of program integrity. Section 7.1 addresses the evaluation of impacts 
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certification error. Section 7.2 discusses the evaluation of impacts on cashier error in identifying 
reimbursable meals. Section 7.3 presents the evaluation of impacts on meal counting and claiming 
error at the school and LEA levels. Section 7.4 summarizes the overall findings on the program 
integrity impacts of the CEP. 

7.1 Administrative Errors Associated with the Certification Process 

This section examines the impact of the CEP on errors in the certification of students for FRP meals. 
These errors occur when LEAs do not comply with requirements for processing applications for FRP 
meals or for identifying students as eligible for free meals without applications. (These “identified 
students” are approved based on direct certification or other list-based certification, e.g., lists of foster 
children.) 112 The basic requirements for correct certification are (1) adequate documentation of 
eligibility, and (2) correct FRP status determination (from this documentation) based on program 
eligibility criteria. When either requirement is not met, there is an administrative error: the LEA has 
incorrectly administered the program rules. When the second requirement is not met, the FRP status is 
incorrect, and a certification error has occurred. 

Not all administrative errors result in certification errors. For example, a student might be approved 
for FRP meals based on an application considered incomplete because an adult household member is 
listed without providing income or an indication of zero income. This represents an administrative 
error. Flowever, if the income shown on the application is consistent with the LEA’s determination of 
the student’s eligibility status, then there is not clear evidence of a certification error. One plausible 
interpretation of the data in this scenario is that the LEA could have confirmed that the household 
member had no income, but the LEA failed to record this on the application. Likewise, the inability of 
an LEA to locate documentation supporting a student’s direct certification for free meals constitutes 
an administrative error, but does not in itself provide evidence that the student was wrongly certified. 
In this chapter, administrative errors that do not clearly represent certification errors are considered 
procedural errors (i.e., a flaw in the procedure but not in the result). 

The analysis presented in this section distinguishes between certification and procedural errors. There 
are three types of certification errors: overcertification (when students receive higher benefit levels 
than their correct eligibility status indicates), undercertification of approved students (when students 
receive lower benefit levels than their correct eligibility status indicates), and denial of benefits to 
eligible students (another form of undercertification). When the review of applications and other 
certification records identified an administrative error, absent clear evidence of certification error, 
such errors were considered procedural.. 

This chapter does not consider errors that occur when households misreport information used to 
determine their eligibility for FRP meals. Such errors can result in either over- or under-certification. 
By eliminating FRP meals applications, the CEP also eliminates these types of errors. Nor does the 
chapter consider undercertification errors when the LEA fails to identify students as eligible when 
they appear on SNAP, TANF, or other lists used for certification without application. Determination 
of household reporting error and students missed in the identification of students from SNAP, TANF, 
or other lists would require intensive data collection beyond the scope of the current study. Thus, the 


112 Other lists used to identify students eligible for free meals include homeless, migrant, runaway, and Head 
Start children/youth. 
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results may understate the impact of the CEP on total certification errors. The discussion of results 
considers relevant information on household reporting errors from the most recent study conducted 
for FNS (Ponza et al., 2007). As a further caution, the sample for this study is not nationally 
representative, and the rates of certification and procedural errors reported herein should considered 
only for the purposes of assessing the impact of the CEP on errors in LEAs likely to take up the 
CEP. 

7.1.1 Expectations for CEP Impacts on Certification and Procedural Errors 

For schools operating under the CEP, application processing errors disappear because no applications 
are processed. The CEP impact on both certification and procedural errors associated with 
applications can therefore be expected to be a 100 percent reduction in CEP schools. (LEAs with a 
mix of CEP and non-CEP schools may have application errors in non-CEP schools). Certification and 
procedural errors associated with identified students may also disappear for some LEAs because, after 
establishing their claiming percentages in the first year of implementing the CEP, they are not 
required to conduct direct certification for four years. Elowever, they may continue to conduct direct 
certification in the hope that they may achieve higher ISPs and claiming percentages. 113 If LEAs 
continue to conduct direct certification and other processes for identifying students as eligible for free 
meals based on other lists, errors in this “student identification” process may rise or fall. 114 Results 
from the previous chapter suggest that participating LEAs usually maintained direct certification 
processes. Thus, the expected direction of impacts on certification and procedural errors for 
identified students is uncertain. Prior research (Ponza et al., 2007) indicates that the the types of 
administrative errors unique to applications are more common than the errors that occur both for 
applications and for identified students, so impacts on total error should be driven mostly by the 
elimination of errors associated with applications. 

7.1.2 Outcome Measures 

Two broad types of error were measured, as discussed above: certification error and procedural error. 
Certification errors - those which resulted in a student being certified for the wrong level of FRP 
meals eligibility - were: 

• Undercertification 

Undercertification occurs when a student is certified to receive a lower level of FRP benefits than 
supported by the application or other documentation. For example, if a student has been denied 
FRP meals but has an application that supports the student’s eligibility for free or RP meals, the 
student has been undercertified. Applications approved for RP meals when the application 


113 Direct certification is required for non-CEP LEAs and recommended but not mandatory for CEP LEAs 
during the four-year cycle of CEP participation. CEP LEAs must conduct direct certification at least every 
four years to establish claiming percentages. As discussed in Chapter 3, CEP LEAs may choose to or be 
required to conduct direct certification in order to provide data on student poverty for use in educational 
programs. 

114 Errors would rise if LEAs seek to identify more students as eligible for free meals at the expense of 
accuracy. Errors would fall if LEAs shift resources from application processing to identifying students 
eligible for free meals based on SNAP and other lists, and therefore conduct the process more thoroughly 
and carefully. 


Abt Associates 


7. Program Integrity Impacts | pg. 129 





Community Eligibility Provision Evaluation: Final Report 


indicates eligibility for free meals also are considered to be undercertified. Identified students are 
eligible for free meals; if identified students are listed as eligible for reduced price rather than free 
meals, they too are undercertified. Only applications with complete income reporting were 
flagged as undercertifications. 115 

• Overcertification 

Overcertification occurs when an application is approved for a higher level of benefits than 
supported by the information contained in the application. For example, if a student is approved 
for free meals, but the income reported on the application is too high (for the household size) for 
the student to be eligible, this constitutes an overcertification error. An approved application that 
is incomplete might be considered a case of overcertification. Flowever, since this situation does 
not provide definite evidence of overcertification, it is considered a procedural error for the 
puiposes of this study. Identified students are eligible for free meals and cannot be overcertified. 

The analysis also measured two kinds of procedural errors for identified students and seven kinds 
for applications. As discussed above, procedural errors occur when the LEA does not maintain 
proper documentation of certifications, but there is not clear evidence that the LEA made a 
certification error. The reader is cautioned that these errors should not be construed as resulting from 
an audit, and are likely to overstate actual error. LEAs were under no legal compulsion to comply 
with this evaluation’s data collectors, and apparent procedural errors might have been resolved had 
LEAs expended the effort to do so. 116 

For identified students in treatment and comparison schools, the categories of procedural errors were: 

• No supporting documentation for free-eligible without application 

This error occurred when a student name appeared on a school’s list of directly certified or other 
identified students, and no supporting documentation could be found. That is, the student could 
not be found on direct certification or other lists that would qualify the student for free meals. 

• Mismatched identifiers between lists of students approved for FRP meals and supporting 
documents. 

This error occurred when documentation existed to support a given student’s FRP eligibility 
without application, but there was reasonable doubt about whether the documentation was for the 
right student. This represents a procedural error because the documentation does not clearly 
support the eligibility determination. Mismatched names were the most co mm on error. The 
remaining cases had matched names, but mismatched birth dates or grades. 


115 Incomplete income reporting on documents available to the evaluation team results in total income 

computations that are too low, because missing items are treated as zero. Flowever, due to the uncertainty 
about the missing information, these cases are treated as procedural errors, not undercertification. 

11(1 The analysis of procedural errors for applications focused on those errors that might be interpreted as 
indicating the presence of a certification error. Other violations of rules, such as a missing signature or 
Social Security Number, were observed but are not included in the analysis here. 
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In addition, for comparison schools, rates were measured for the following procedural errors for 
applications: 

• Application missing or no eligibility determination could be made based on the application 

This error occurred when a student was listed as being certified for FRP based on an application, 
but no application was found, or the information on the application was insufficient to compute 
FRP eligibility. 

• Mismatched identifiers between FRP certification lists and supporting documents 

This error occurred when an application was found but there was reasonable doubt about whether 
the application or the LEA’s determination was for the right student. As with identified students, 
mismatched names were the most co mm on error. The remaining cases had matched names, but 
mismatched birth dates or grades. 

• TANF or SNAP case number has the incorrect number of digits (for categorically certified 
applications only) 

An application can be approved based on providing a case number for TANF or SNAP, instead of 
providing household income. The number of digits in the case number on the application was 
compared to the valid number for the State to determine if the case number was valid. 117 

• Certified before start of the school year (July 1, 2012) 

Students eligible by application must submit an application and be certified for FRP meals every 
school year. If a student was certified for FRP meals for the 2012-13 school before July 1, 2012, 
it suggests that the certification from the prior school year was used. 

• Income is the basis of eligibility determination, but no income reported 

This error occurred if the LEA indicated that the student was certified on the basis of income, but 
no income was reported on the application. Note that if the household reported an income of 
zero, the application would not be considered to have an error. 

• Income not reported for at least one household member whose income should be counted. 

If an individual is listed on the household roster section of the application, either “no income” for 
that person must be indicated or an amount must be provided. If neither was provided on an 
approved application, it was counted as a procedural error. 118 

• An amount for income is reported, but the associated time period is missing. 


117 This type of error has not been treated as a certification error in the Application, Participation, Eligibility, 
and Certification (APEC) study (Ponza et al., 2007) or FNS reviews of applications, so it is treated as a 
procedural error. 

118 The application form used by the LEA may not clearly indicate to the household that the lack of income 
should be reported. In such cases, non-reporting of zero income might not be considered a procedural error. 
However, the lack of this information is inconsistent with the FNS Eligibility Manual for School Meals. 
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For any amount of income reported on the application, an associated time period (e.g., weekly, 
monthly) must also be reported. If no time period was reported, it was counted as a procedural 


7.1.3 Data and Methods 

As described in Chapter 2 and further explained in Appendix 7A, certification records and FRP 
applications were sampled and collected from schools within the Component 2 sample of treatment 
and comparison LEAs (see Exhibit 7.1). 120 The analysis file comprised data on 248 schools (119 
treatment and 129 comparison) in 102 LEAs (51 treatment and 51 comparison). 121 A total of 9,257 
records of identified students were sampled (5,401 from treatment schools and 3,856 from 
comparison schools). These were students who did not apply for meal benefits but were directly 
certified from SNAP, TANF, or other lists of eligible households. In addition, 2,920 FRP 
applications were sampled from comparison schools. Of these applications, 1,764 were approved and 
1,156 were denied. 

The methods used to analyze certification and application records were necessarily different than 
those used for other impact analyses, due to the particular nature of the data. That is, there were no 
applications and hence no application errors for CEP schools. Thus, no analysis is required to 
estimate the program’s impact on these errors. Whatever those errors are, the CEP will reduce them 
to zero. The “impact” will therefore vary from sample to sample—and indeed from school to 
school—in known and predictable fashion. This report shows what those errors were in the study 
comparison schools; as noted above, however, these schools were not selected in such a way to be 
representative of any national or State population of schools. 


Exhibit 7.1: Certification Records Review Sample Sizes 



Treatment 

Comparison 

Number of LEAs 

51 

51 

Number of schools 

119 

129 

Number of certification records 

Direct Certifications 

5,290 

3,770 

Other Free Eligibility List 

111 

86 

Approved applications 


1,764 

Denied applications 


1,156 

Total 

5,401 

6,776 


Source: Certification Record and Application Reviews. 


For identified students, the analysis estimates CEP impacts on procedural errors using the model for 
the posttest-only comparison group design from Chapter 2; the form of the estimation model for these 


119 LEAs may specify on their application that income should be reported on a monthly basis, or that income 
will be assumed to be on a weekly basis unless otherwise specified. 

120 Complete details on the construction of the certification error analysis file are/will be included in the 
technical specifications accompanying the analytic files. 

121 All treatment group schools in the sample were CEP schools and therefore did not collect applications. 
LEAs in the treatment group sample had at least 70 percent of schools with the CEP. For those treatment 
LEAs with some non-CEP schools, the results represent impacts on the CEP part of the LEA. 
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specific analyses is described in Chapter 6. Only impacts on procedural error were estimable - for 
reasons detailed below. 

Appendix 7A presents the characteristics of the analysis sample. In brief, treatment and comparison 
schools and LEAs were statistically indistinguishable on 23 of the 26 variables tested, with three 
exceptions: enrollment (treatment schools had fewer students, on average), FRP eligibility (treatment 
schools had higher average FRP eligibility), and the LEA’s ISP (treatment LEAs had higher average 
ISPs). The analytic model controlled for these differences, as explained in Appendix 7A. 

7.1.4 Impacts on Schools’ Certification Error 

Certification error consists of under- or overcertification. For identified students, overcertification is 
infeasible (they are all eligible for free meals) and undercertification is extremely rare. Only 7 of the 
3,770 identified students in comparison schools - and none of the 5,290 identified students in CEP 
schools - were certified as eligible for reduced-price, not free, meals. This suggests that there may 
have been an impact on these extremely rare errors, but it was not statistically estimable because the 
outcome measure for all treatment schools was zero. Thus, the analysis of certification error focused 
on approved and denied applications, and the objective was to estimate error rates for these 
applications. 

Exhibit 7.2 shows certification error rates for applications in the sample of comparison schools. For 
approved applications, the school average undercertification percentage was 1.5 percent, and the 
overcertification percentage was 5.0 percent. The school average undercertification rate for denied 
applications was 8.3 percent. By definition, denied applicants cannot be overcertified. 

The total certification error rate for all applications cannot be computed for the study sample, because 
denied applications were disproportionately sampled, and not all schools provided the total numbers 
of approved and denied applications required to weight the data appropriately to obtain school-level 
estimates. Instead, Exhibit 7.2 presents an illustrative total error estimate based on the sample median 
ratio of denied to total applications for the sampled schools that provided these data. This procedure 
yields school-average under- and overcertification rates of 2.1 and 4.6 percent, respectively. Thus, 
the rates shown in Exhibit 7.2 roughly suggest the magnitude of potential reduction in certification 
error for applications that may be associated with CEP implementation . Although the sample for this 
study is not nationally representative, these figures are roughly comparable to the under- and 
overcertification rates of 2.1 and 6.2 percent found for certified students and denied applicants 
(combined) in the SY 2005-06 Application, Participation, Eligibility, and Certification (APEC) study 
(Ponza et al., 2007). 122 


122 The estimates provided here from Ponza et al. (2007) are for administrative errors and do not include 
household reporting errors. Ponza et al. do not report separate administrative error rates for approved 
applications versus identified students, so these figures understate the truly comparable error rates. 
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Exhibit 7.2: Average School Certification Error Rates for Applications 



Approved Applications 

Denied Applications 

Total 3 

Undercertified 

1.5 

8.3 

2.1 

Overcertified 

5.0 

0.0 

4.6 

Total 

6.5 

8.3 

6.6 

Number of schools 

127 

128 


Number of LEAs 

51 

51 



Computation of total assumes 8 percent of applications are denied; in this sample of schools, 8 percent is the 
median percentage of applications that were denied. 

Source: Certification Record and Application Reviews. 


7.1.5 Impacts on Schools’ Procedural Errors 

Procedural errors are those that violate program requirements but do not provide sufficient 
information to determine that the student received the wrong FRP certification. Exhibit 7.3 shows 
estimated impacts of the CEP on two kinds of procedural errors for identified students. Results are 
displayed as regression-adjusted means for the treatment group, actual means for the comparison 
group, differences in means, and percentage differences in means. Lack of supporting documentation 
(when the student could not be found on a direct certification list or other document substantiating 
categorical eligibility) was the most prevalent error, occurring for 3.0 percent of identified students in 
comparison schools and 2.1 percent of identified students in treatment schools. In both comparison 
and treatment schools, 1.2 percent of identified students had mismatched identifiers between FRP 
certification lists and supporting documents. Overall, the CEP was associated with a 1 percentage 
point reduction in certification error for identified students, from 4.3 percent in comparison schools to 
a regression-adjusted 3.3 percent in treatment schools. This impact was not statistically significant, 
however. 


Exhibit 7.3: CEP Impacts on School Procedural Error Rates for Identified Students 



Comparison 

Treatment 

T-C Difference 

Percent 


Schools 

Schools 

(Impact) 

Impact 

No supporting documentation for 
free-eligible without application 

3.0 

2.1 

-1.0 

-32.3 

Mismatched identifiers between 

FRP certification lists and 
supporting documents 

1.2 

1.2 

0.1 

6.3 

Any procedural error 

4.3 

3.3 

-1.0 

-23.7 

Number of schools 

129 

119 



Number of LEAs 

51 

51 




***=p<0.01; **=p<0.05; *= p<0.10. 

Source: Certification Record and Application Reviews. Means for treatment schools are regression-adjusted for 
differences in sample characteristics. 


Exhibit 7.4 reports on the actual error rates for various types of procedural errors for both approved 
and denied applications. As in Exhibit 7.2, an illustrative total application error estimate is shown 
based on the sample median ratio of denied to total applications for the sampled schools that provided 
these data. 
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Exhibit 7.4: Average School Procedural Error Rates for Applications 


Approved 

Applications 

Denied 

Applications 

Total 8 

Application missing 

1.6 

1.7 

1.6 

Mismatched identifiers between FRP certification lists 
and supporting documents 

3.0 

4.0 

3.1 

TANF or SNAP case number has the incorrect 
number of digits (categorically certified only) 

1.6 

0.0 

1.5 

Certified before 1 July 2012 

0.6 

0.8 

0.6 

Income is the basis of eligibility determination, but no 
income reported 

1.5 

1.4 

1.5 

Income not reported for at least one household 
member whose income should be counted. 

9.7 

7.5 

9.5 

An amount for income is reported, but the associated 
time period is missing. 

3.4 

3.8 

3.4 

Any procedural error 

20.2 

16.9 

19.9 

Number of schools 

127 

128 


Number of LEAs 

51 

51 



a Computation of total assumes 8 percent of applications are denied; in this sample of schools, 8 percent is the 
median percentage of applications that are denied. 

Source: Certification Record and Application Reviews. 


Overall, about 20 percent of approved applications and 17 percent of denied applications had some 
sort of procedural error. Error rates for approved and denied applications were, in most cases, within 
roughly one percent of one another. Applications were missing in about 1.6 percent of both approved 
and denied applications, and mismatched identifiers occured in 3.0 to 4.0 percent of applications. 
About 1.6 percent of approved applications had the wrong number of digits in the TANF or SNAP 
case number provided to support categorical eligibility; this error never occurred for denied 
applications because no denied applications were based on categorical eligibility. The prevalence of 
income reporting errors was similar on approved and denied applications, although approved 
applications had a somewhat higher percentage of applications where income was not reported for at 
least one household member whose income should be counted (9.7 versus 7.5 percent). 


7.2 Cashier Errors 

This section assesses the effects of the CEP on cashier error rates. Using the posttest-only comparison 
design described in Chapter 2 as background, this section (1) discusses expectations about why and 
how effects might arise; (2) presents the specific outcome measures used; (3) provides more detail 
about data collected; and (4) describes the specifics of the model used for analysis, and (5) presents 
the results about cashier error rate. 

7.2.1 Expectations about CEP Impacts on Cashier Errors 

There are several steps in the process of counting and claiming meals served under the NSLP and 
SBP. The first step is cashier determination and entry of whether a meal is eligible, based on (a) 
whether it is taken by a student, and (b) whether the student has taken the required meal components. 
In later steps, the school aggregates these counts and reports them to the LEA central foodservice 
office, and the LEA submits the claim for reimbursement with meal counts. This section discusses the 
error rate for the first step, and Section 7.3 discusses the error rates for the remaining steps. 
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Counting errors occur when cashiers make mistakes in determining whether specific meals are 
reimbursable. Under the CEP, counting and claiming errors could rise or fall. Under conventional 
meal counting procedures, the cashier determines whether the student is approved for free or reduced 
price meals, as well as whether the meal is reimbursable. The error rate may fall as the checkout 
process is simplified by the elimination of distinctions based on certification status, which also may 
lead to fewer errors in the process of aggregating cashiers’ determinations under the CEP. Also, if 
menus are simplified or alternate serving methods are used under CEP, these could reduce cashier 
errors. In particular, if all students are served the same meal (as is sometimes the case, particularly 
when breakfast is served in the classroom), then the cashier does not need to determine whether each 
individual meal is reimbursable. On the other hand, cashiers might make more errors if LEAs increase 
the variety of their menus as a result of the CEP, or if LEAs respond to an increase in meal 
participation by requiring the same number of cashiers to work faster. 

As noted in Chapter 2, the evaluation addressed the following exploratory research question on this 
topic area: 


• How did the CEP impact errors in meal counting by cashiers? 


7.2.2 Outcome Measures 

As with certification error, the analysis of cashier error is based on an algorithm for determining when 
errors occur. Specifically, two types of errors are identified through analysis of the observation data 
discussed in the next section: over-claiming (counting meals as reimbursable when they are not) and 
under-claiming (not counting meals as reimbursable when they are). The data included a 
determination of the correct status of each meal based on the items selected by the observed student. 
The cashier error rate is the sum of the over-claiming and under-claiming error rate. 123 All three rates 
were calculated separately for NSLP and SBP. 


Overclaiming error rate 


Number of Non-Reimbursable Meals counted as Reimbursable 
Total Number of Meals 


(Eq. 7.1) 


Underclaiming error rate 


Number of Reimbursable Meals counted as Non-Reimbursable 
Total Number of Meals 


(Eq. 7.2) 


Cashier error rate = Overclaiming error rate + Underclaiming error rate (Eq. 7.3) 

7.2.3 Data and Final Analytic Sample 

The impact of the CEP was examined by comparing the cashier error rates in counting the meals 
(lunch or breakfast) after the intervention was implemented in these schools to a matched group of 
non-participating schools at the same time point. As discussed in Chapter 2, a systematic subsample 
of 52 LEAs was sampled for Component 3; the sample includes 27 treatment (participating) LEAs 
matched with 25 comparison (non-participating) LEAs using a propensity score. 124 Within these 
LEAs, 81 treatment and 75 comparison schools were sampled. Exhibit 7.5 presents the final analytic 


123 Overclaims represent a cost to the taxpayer. Underclaims represent a cost to the child who must pay the 
price of a non-reimbursable meal; the cost is the difference between the actual price paid and what the child 
would have paid if the meal had been classified as reimbursable (which could be zero or a fraction of the 
non-reimbursable meal price). 

124 The count of comparison LEAs excludes two LEAs subsequently found to be ineligible. 
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samples of LEAs, schools, and cashier observations. (Observation procedures are discussed below.) 
The table notes the reasons for deviations from the original sample of schools. Appendix 7B shows 
the difference between the LEA and school characteristics of the applicable sample of treatment and 
comparison schools. In brief, treatment and comparison schools were statistically indistinguishable on 
all LEA-level covariates tested (as the matching was done at the LEA level). Out of the 11 school 
level characteristics, the differences between treatment and comparison schools were statistically 
significant for the percentage of students who qualified for free or reduced price meals in the school, 
whether a school was Title I, whether the school was a charter school, and the percentage of students 
who were Black in the school. These characteristics were used in the analytic model (presented in the 
next section) to control for potential bias due to imbalance; State indicators also were included in the 
model. 


Exhibit 7.5: Sample for Cashier Observations 



Treatment 

Comparison 

Total 

For NSLP 

Matched LEA sample for cashier observations 

27 

25 

52 

Schools for cashier observations 

80 

75 

155 

Cashier observations 

5,300 

4,916 

10,216 

For SBP 

Matched LEA sample for cashier observations 

27 

25 

52 

Schools for cashier observations 

75 

73 

148 

Cashier observations 

3,225 

3,288 

6,513 


See Appendix 7B for discussion of sample. 


As described in Chapter 2, the data collection for cashier observations was conducted in conjunction 
with the menu survey collection. 125 Using the menu survey, coders decided whether a meal selected 
by a student in each observation was reimbursable. 126 Error rates for cashier determinations were 
computed for each school separately for breakfast and lunch. The analysis compared the resulting 
cashier error rates between the treatment and comparison groups. 

7.2.4 Analytic Model 

The posttest-only comparison group design was used to assess the impact of the CEP on errors in 
meal counting by cashiers. As explained in Chapter 2, this design compares the treatment group with 
a matched comparison group after the treatment has been implemented. The comparison group’s 
mean outcome represents the closest possible counterfactual, that is, what would have happened 
without the CEP, and the analysis compares that to the outcomes observed in CEP schools to produce 
an estimate of the CEP’s impact, after adjusting the treatment group mean for differences in school 


12:1 In each school participating in Menu Survey data collection, observers positioned themselves on cashier 
lines, recording the meal selections of a sample of students, and for each observation, recorded the cashiers’ 
determination of the reimbursable status for the meal. A total of 100 observations were obtained at lunch 
(60) and at breakfast (40) during one day in the week when Menu Survey activities took place. 

126 Coders used two main steps to determine the reimbursable status of each meal observed. First, the meal 
component codes of the food items in each observation were reviewed, and then the information from the 
Menu Survey was used to determine whether a selected food met the requirements for a reimbursable food. 
Second, coders determined whether the number of reimbursable items selected qualifies for the 
reimbursable meal, under offer-versus-serve guidelines, if applicable for the school. 
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characteristics using the parameters from a regression model. The estimation model for posttest-only 
comparisons at the school level is described in Chapter 6. Because the treatment group includes only 
LEAs that take up the CEP in all schools, the effect of participating is the effect of fully participating 
in the treatment; that effect may differ from the effect of take up on only a subset of schools within an 
LEA. 

7.2.5 Cashier Error Impact Results 

As noted above, two sets of cashier error outcomes were tested: one for lunch (NSLP) and one for 
breakfast (SBP). This analysis used the school-level posttest-only comparison group design described 
in the above section. 

Exhibit 7.6 provides the regression-adjusted cashier error means for the treatment and comparison 
schools by school grade level and overall for NSLP. The overall average cashier error was 3.7 percent 
for the comparison schools, and the adjusted mean (controlling for differences in schools) was 2.9 
percent for the treatment schools. The estimated difference of 0.8 percent was not statistically 
different from zero. Thus, the CEP apparently had no overall impact on cashier error for the NSLP. 

The results varied across grade levels. For elementary and middle schools, the results indicated a 
slight increase in the error rate in the treatment group, but this change was not statistically different 
from zero. For high schools, the estimated impact was a 4.3 percentage point decrease in cashier 
errors for NSLP, and this difference was statistically significant as well as large in magnitude: that 4.3 
point decrease represents a 60 percent relative decrease in errors. Additional details of the analysis are 
presented in Appendix 7B. It appears that the overall lack of significant impact on cashier errors at 
lunch reflects the lack of impact at the elementary and middle school levels, offsetting the impact at 
the high school level. 

Exhibit 7.6: Impact of CEP on Cashier Error Rate in National School Lunch Program by Grade 
Level 



Comparison 

Schools 

Treatment 

Schools 

T-C Difference 
(Impact) 

Percent 

Impact 

Elementary schools 

2.05% 

3.17% 

1.1% 

54.8% 

Middle schools 

2.15 

3.30 

1.2 

53.5 

High schools 

7.15 

2.88 

-4.3* 

-59.7 

All schools 

3.72 

2.92 

-0.8 

-22.8 

Number of Schools 

155 




Number of LEAs 

52 





Means for comparison group are unadjusted (actual means). Means for treatment group are regression-adjusted 
to control for differences between treatment and comparison schools. The regression with covariates results 
appear in Appendix 7. Information from the first three rows comes from one regression and the information from 
the last row comes from a separate regression that combines all grades. 

***=p<0.01; **=p<0.05; *=p<0.10. 

Source: Cashier Observation and Menu Survey Data. 

The same methods were used to estimate the CEP’s impact on cashier counting errors for breakfast. 
Exhibit 7.7 displays the regression-adjusted cashier error means for the treatment and comparison 
schools by school grade level and overall for SBP. The average cashier error rate for treatment and 
comparison schools was not statistically different (5.3 and 6.4 percent, respectively). As was the case 
for lunch meals, the results varied across grade levels. For middle and high schools, there was no 
significant impact on the error rate. For elementary schools, however, the estimated impact was a 4.9 
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percent decrease in cashier errors for meal counting during breakfast, representing a statistically 
significant and large, 73 percent relative decrease in the error rate. The shift to classroom breakfast 
(as discussed in Chapter 5) in the CEP schools may have been a contributing factor in this impact. As 
was the case for the NSLP, the impact in one category of schools was offset by a lack of impact in the 
other two, resulting an overall finding of no impact. Additional details of the analysis are presented in 
Appendix 7B. 


Exhibit 7.7: Impact of CEP on Cashier Error Rate in School Breakfast Program by Grade Level 



Comparison 

Schools 

Treatment 

Schools 

T-C Difference 
(Impact) 

Percent 

Impact 

Elementary schools 

6.75% 

1.85% 

-4.9%“ 

-72.6% 

Middle schools 

5.80 

5.88 

0.1 

-1.5 

High schools 

6.62 

8.45 

1.8 

27.6 

All schools 

6.40 

5.25 

-1.1 

-18.0 

Number of schools 

148 




Number of LEAs 

52 





Information from the first three rows comes from one regression and the information from the last row comes 
from a separate regression that combines all grades. The results are from the regression with covariates and 
appear in Appendix 7B. 

***=p<0.01; “=p<0.05; *=p<0.10. 

Source: Cashier Observation and Menu Survey Data. 

7.2.6 Discussion of the Results 

The CEP’s impact on cashier error rate was negligible for both the NSLP and SBP. Though cashier 
error rates appeared to decrease modestly across both outcomes, the differences were generally not 
statistically significant (except for high school for lunch and elementary school for breakfast). These 
estimates should be interpreted carefully. As discussed in the introduction to this chapter, these 
estimates are specific to the group of schools that were observed, and they are based on a relatively 
weak evaluation design using posttest-only comparisons. 


7.3 LEA Claiming Errors 

This section assesses the effects of the CEP on LEA claiming error rates. Using the posttest-only 
comparison design detail described in Chapter 2 as background, this section (1) discusses 
expectations about why and how effects might arise; (2) presents the specific outcome measures used; 
(3) provides more detail about data collected; and (4) describes the specifics of the model used for 
analysis, and (5) presents the results about claiming error rates. 

7.3.1 Expectations about CEP Impacts on Claiming Errors 

As discussed in the introduction to this Chapter, the evaluation addressed the following exploratory 
research question regarding claiming error: 

• Plow did the CEP impact errors in computing claims for reimbursement? 

As noted in the previous section, there are several steps in the process of counting and claiming meals 
served under the NSLP and SBP. The first step is cashier entry and determination of whether a meal 
is eligible, based on (a) whether it is taken by a student, and (b) whether the student has taken the 
required meal components. The second step is aggregating these counts and reporting to the LEA 


Abt Associates 


7. Program Integrity Impacts | pg. 139 













Community Eligibility Provision Evaluation: Final Report 


central foodservice office. The third step is submitting the claim for reimbursement with meal counts 
to the State. Claims in the study States are submitted on a monthly basis, so daily totals for schools 
must be summed. Depending on the State, claims may be submitted for individual schools or the 
entire LEA. Errors in the second and third steps are referred to as claiming errors. This section 
discusses the claiming error rate; the previous section discussed the error rate for the first step. 

Claiming errors can occur at three points. First, errors may occur at the individual school level when 
consolidating the different cashier totals or adding up tally sheets in a school. Second, there may be 
an error when the meal counts reported by the school are recorded by the LEA foodservice office. 
Third, errors may occur when the LEA computes the totals for the claim period, either for individual 
schools or for the LEA (if totals for schools are consolidated into a single LEA claim). 

Under the CEP, claiming errors in the first two parts of the claiming process were expected to fall. At 
the school level, the CEP simplifies the claiming process, because the school only needs to compute 
and record the total meals served for NSLP and SBP. Under conventional claiming procedures, 
schools must compute and record totals for free, reduced-price, and paid meals. Reporting from the 
school to the LEA is simplified in the same way, so there is less potential for error. 

In the third stage of the claiming process, the CEP could reduce or increase error. On the one hand, 
computations use only the total meal counts from the schools, so in one way the process is simpler 
than conventional claiming. On the other hand, the CEP introduces a new step in the process of 
submitting the claim to the State: the LEA must compute the counts of meals to be claimed at the free 
and paid levels, using the appropriate claiming percentage. As discussed in Chapter 1, the claiming 
percentage is based on the ISP for the school, a group of schools, or the LEA, depending on how the 
LEA chooses to submit its claims. If the LEA uses the wrong claiming percentage or makes an error 
in the computation of the free and paid meal counts, the claim will be wrong. On the other hand, if the 
claiming system automatically provides the correct claiming percentage and computes the free and 
paid meal counts, there will be no new error, and overall, the expected result would be lower error. 

7.3.2 Outcome Measures 

The analysis of claiming error examined errors rates for claims for NSLP, SBP, and the two programs 
combined. These errors were defined in two ways: signed and unsigned. With signed errors, a 
positive sign indicates that the amount claimed for a school was more than the LEA was entitled to 
(i.e., an overclaim), and a negative sign indicates that the claim was less than the LEA was entitled to 
(an underclaim). An overclaim represents a loss to USDA; an underclaim represents a loss to the 
LEA. Thus, an impact on signed errors would represent a shift of funds, on average, between LEAs 
and USDA. 

An unsigned error is the absolute value of a signed error. The reason for using unsigned errors is that 
there may be concern about whether the overall accuracy of claims has increased. It is possible that a 
group of LEAs and schools could have small average signed claiming errors while having substantial 
average unsigned claiming errors: this could occur if over-claim errors approximately offset under¬ 
claim errors within the group. Since average signed errors can therefore give an incomplete picture of 
the extent of claiming errors, it was useful to also examine average unsigned errors. 

The data analyses reported below used error rates rather than errors (in dollars) as outcomes. For a 
specific meal program (NSLP or SBP), a school’s signed error rate is 
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dollar amount claimed by the LEA for the school-dollar amount to which it is entitled 
dollar amount claimed by the LEA for the school 


(Eq. 7.4) 


For example, a school’s signed error rate for the NSLP is the signed NSLP claiming error (in dollars) 
divided by the dollar amount that the LEA claimed for that school for the NSLP. A positive signed 
claiming error rate of 5 percent would indicate that the LEA was overpaid by 5 percent. For a specific 
meal, a school’s unsigned error rate is the absolute value of its associated signed error rate, so a 
negative signed error rate becomes positive as an unsigned error rate. 


A school’s total signed error rate is 

total dollar amount claimed by the LEA for the school-total dollar amount to which it is entitled 
total dollar amount claimed by the LEA for the school 


(Eq. 7.5) 


where total dollar amount claimed is the sum of the dollar amounts claimed for breakfast and for 
lunch, and the total dollar amount to which the LEA is entitled is the sum of the dollar amounts for 
breakfast and for lunch to which the school is entitled. A school’s total unsigned error rate is the 
absolute value of its total signed error rate. 


In summary, the six outcome measures were: 


• Signed error rates for NSLP, SBP and combined programs 

• Unsigned error rates for NSLP, SBP, and combined programs 

7.3.3 Data and Final Analytic Sample 

The impact of the CEP on participating schools was examined by comparing the six claiming error 
rates after the intervention was implemented in these schools to a matched comparison group of non¬ 
participating schools. As discussed in Chapter 2, a systematic subsample of 52 LEAs was selected for 
Component 3, matching 27 participating “treatment” LEAs with 25 non-participating “comparison” 
LEAs using propensity scores. 127 Three schools were sampled in each LEA to represent the 
elementary, middle and high school grade levels. 

Exhibit 7.8 presents the final analytic sample for the claiming errors. The matching was done at the 
LEA level. Appendix 7C shows the difference between the LEA and school characteristics of the 
applicable sample of participating and non-participating schools, and discusses differences between 
the overall Component 3 sample and the sample for this analysis. The treatment and comparison 
groups were well-balanced on all LEA characteristics except number of schools, but there were 
several school characteristics that showed imbalance: urban, percent Black, percent Hispanic, charter 
school, percent FRP. These unbalanced characteristics were used as predictors in the analytic model 
(discussed in the next section) to control for potential bias due to the imbalance on these factors. 


127 Two LEAs, both comparison group members, were found to be ineligible for the Impact Study after 
completion of the data collection. 
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Exhibit 7.8: Sample Design for Community Eligibility Option Evaluation Sample Design for 

Community Eligibility Option Evaluation 




Treatment 

Comparison Total 

Matched LEA sample for claiming errors 

24 

23 47 

Schools for claiming errors 

70 

67 137 


As stated in Chapter 2, the data collection for assessing claiming errors was conducted by field 
interviewers during site visits to schools and to LEAs. During site visits to schools, field interviewers 
recorded the numbers of SBP and NSLP meals of each category (free, reduced, price, and paid) 
according to school records and separately verified these data using the supporting documentation 
that the school used to produce meal counts. The school counts from school records and the verified 
counts were for a single specified school day. During site visits to LEAs, field interviewers obtained 
LEA records of meal counts from the schools, as well as the number of meals claimed for 
reimbursement by the LEA. The LEA records of school meal counts were both for the day specified 
for the school visit data collection and for an entire reporting period; the meals claimed were for the 
entire reporting period. For CEP schools, data collectors obtained the claiming percentage used for 
each school (or for the entire LEA, if school counts were consolidated before computing free and paid 
meals). The correct claiming percentage for each school/LEA was obtained from State administrative 
data. 

7.3.4 Analytic Model 

The posttest-only comparison group design was used to assess the impact of the CEP on claiming 
errors. As explained in Chapter 2, in this design the treatment group is compared with a comparison 
group after the treatment has been implemented. 

For claiming errors, the outcome was measured at the school level and was likely to vary by grade 
level. The analysis was, therefore, done at the school level using both school- and LEA-level control 
variables. The regression models to estimate impacts followed the form discussed in Chapter 6. The 
models included the treatment indicator, school grade level, and State, as well as the school 
characteristics that were unbalanced between treatment and control groups (as discussed above). To 
estimate impacts for each school grade level, the regression models included interactions of treatment 
with grade level. The regression model specification and results are presented in Appendix 7C. These 
results were used to compute the regression-adjusted mean error rates for the treatment group, as 
presented in the tables in the next section. 

7.3.5 Claiming Error Results 

Exhibit 7.9 presents the estimated impacts of the CEP on signed error rates for the NSLP, SBP, and 
combined programs. As discussed above, signed error rates are positive if the LEA overclaims, and 
negative if the LEA underclaims. 

The results in Exhibit 7.9 follow a consistent pattern: treatment schools had negative signed error 
rates on the order of -1 to -2 percent (after the regression adjustment), while comparison schools had 
positive signed error rates of less than 1 percent overall and for elementary and middle schools, and 
close to 2 percent for NSLP and overall among high schools. Thus, treatment group schools tended to 
underclaim, while comparison group schools overclaimed, and the error rates were quite small. The 
results for all schools suggest that the CEP may have had a negative impact on signed error rates for 
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NSLP, SBP, and combined programs, but these results were not statistically significant at the 5 
percent level. . 


Exhibit 7.9: Impact of CEP on Signed Claiming Error Rated in NSLP, SBP, and Combined 
Programs (by School Type and Pooled Across Schools) 



Comparison 

Treatment 

T-C 

Difference 

(Impact) 

NSLP Impacts 

Elementary schools 

0.0 

-1.2 

-1.2 

Middle schools 

0.1 

-1.2 

-1.2 

High schools 

1.8 

-2.5 

-4.2 

All schools 

0.6 

-1.6 

-2.2* 

SBP Impacts 

Elementary schools 

0.0 

-0.6 

-0.6 

Middle schools 

0.0 

-1.5 

-1.5 

High schools 

0.1 

-2.3 

-2.4 

All schools 

0.1 

-1.4 

-1.4* 

Combined Impacts 

Elementary schools 

0.0 

-1.0 

-1.0 

Middle schools 

0.1 

-1.2 

-1.3 

High schools 

1.6 

-2.4 

-4.0 

All schools 

0.6 

-1.5 

-2.0* 

Number of Schools 

70 

67 


Number of LEAs 

24 

23 



Treatment group means are adjusted to represent expected results in comparison schools if they participated in 
CEP. Estimates for grade-level and total impacts are obtained using slightly different models. Grade-level 
estimates are derived by including interactions between grade-level and treatment status in the model. The 
results are from the regressions are presented in Appendix 7C. 

***=p<0.01; **=p<0.05; *=p<0.10. 

Source: Meal counting and claiming data collection. 

Exhibit 7. 10 presents the estimated impacts of the CEP on the unsigned error rates for the NSLP, 
SBP, and combined programs. The unsigned error rates for the treatment group (after the regression 
adjustment) were between 3 and 5 percent, depending on the program and the grade level, while the 
unsigned error rates for the comparison group were less than 1 percent overall and less than 2 percent 
in all grade levels. The results indicate that CEP increased the unsigned SBP error rate for all schools 
(the result is significant at the 5 percent level), but did not have a significant effect on this measure 
for NSLP or overall. 
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Exhibit 7.10: Impact of CEP on Unsigned Claiming Error Rate in NSLP, SBP, and Combined 
Programs (by School Type and Pooled Across Schools) 



Number of LEAs 24 23 


Treatment group means are adjusted to represent expected results in comparison schools if they participated in 
CEP. Estimates for grade-level and total impacts are obtained using slightly different models. Grade-level 
estimates are derived by including interactions between grade-level and treatment status in the model. The 
results are from the regressions are presented in Appendix 7C. 

***=p<0.01; **=p<0.05; *=p<0.10. Source: Meal counting and claiming data collection. 

7.3.6 Discussion of the Results 

Examination of the underlying data provides some insight into why the CEP increased the unsigned 
error rates for the SBP and may have also affected signed error rates. For the SBP, only four 
comparison schools had errors, and these were evenly divided between errors at the first stage (within 
school) and at the third stage (claim submitted to the State). In contrast, 14 treatment schools had 
errors, and 12 of these had their errors at the third stage. The same pattern was observed in the 
claiming errors for NSLP. The primary reason for errors in claims submitted for CEP schools to the 
State was that the LEA used the wrong claiming percentage, and usually the claiming percentage used 
was less than the one the LEA was entitled to use. 128 Several CEP States have already implemented 
automated systems to prevent errors in the use of claiming percentages. As this practice becomes 
more widespread, it would presumably eliminate the main source of claiming error observed in CEP 
schools. 

From a larger perspective, the estimates of claiming error are roughly consistent with the findings of 
the APEC study (Ponza et al., 2007), which estimated comparable measures of claiming error at 3.5 
percent of NSLP reimbursements and 6.0 percent of SBP reimbursements. Therefore, there is 


128 This determination was based on claiming percentages provided by the States in the fall of 2012. It is 

possible that some claiming percentages were revised, in which case the amount of error may be overstated. 
Also, the data collection relied on the LEA’s records of the claim. If the State corrected the claim, then 
there would have been no actual over- or under-payment. 
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relatively little room for the CEP to increase accuracy in what already appears to be a highly accurate 
process. 
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8. Nutritional Quality Impacts 


The Healthy Hunger-Free Kids Act (HHFKA) of 2010 revised the nutrition standards for the National 
School Lunch Program (NSLP) and School Breakfast Program (SBP). The final implementing rule 
was issued in January 2012 (Federal Register, 2012); it outlines changes in NSLP and SBP meals to 
align closely with the latest Dietary Guidelines and reflects recommendations from the National 
Academies’ Institute of Medicine (IOM, 2010). Key changes in the meal standards include: (1) 
offering fruit daily at breakfast and lunch; (2) increasing the amount and variety of vegetables at 
lunch; (3) offering more whole grain-rich foods; (4) limiting fluid milk choices to fat-free (unflavored 
or flavored) and unflavored low fat milk; (5) establishing minimum and maximum calorie levels for 
each age/grade group; (6) increasing the emphasis on limiting saturated fat; (7) seeking gradual but 
major reductions in the sodium content; and (8) eliminating trans fat in purchased products and 
ingredients used in school meals. 

The new meal standards require schools to follow one food-based meal pattern for each of three 
age/grade groupings for NSLP and SBP, rather than choosing one of the five previous options using 
food-based or nutrient-standard menu planning. 129 The new meal patterns are expected to supply most 
of the “nutrient targets” identified by the IOM in developing the recommendations. Under the final 
rule, schools are responsible for offering meals that meet the meal pattern with daily and weekly 
minimums for specific food components, as well as specific standards for average calories, saturated 
fat, and sodium, over the five-day school week for each age/grade group. The new meal standards for 
NSLP became effective in SY 2012-13, although the sodium specifications are being phased in 
gradually over a 10 year period with the first of two intermediate targets (Target 1) beginning in SY 
2014-15 for both breakfast and lunch. The new meal standards for SBP are effective as of SY 2013- 
14; the exception is the limits on milk offerings in the SBP, which were implemented in SY 2012-13. 
See Appendix 8A for details of the new meal standards for both the NSLP and SBP that were in place 
for SY 2012-13. 

This chapter describes the impact of the CEP on meal quality in lunches and breakfasts by comparing 
a sample of CEP schools to a matched sample of non-participating schools. This analysis uses the 
relevant meal standards in place at the time of the data collection (Winter and Spring 2013). As in 
Chapters 6 and 7, the analysis uses the posttest-only comparison design, and the results should be 
considered exploratory, not definitive. In particular, Chapter 8 presents a large number of statistical 
tests, some of which could be significant by chance. Another important caveat is that, because all 
LEAs were required to implement the new meal standards for lunch in SY 2012-13, the outcomes 


l2Q Prior to the implementation of the new meal standards, schools could opt to use five different approaches to 
planning menus for reimbursable meals. They had two food-based options: “Traditional” or “Enhanced” 
food-based menu planning options. Although similar in structure, the Enhanced option allowed for more 
weekly servings of bread/grains and larger servings of fruits and vegetables. Two other options were the 
Nutrient Standard Menu Planning (NSMP) and the Assisted Nutrient Standard Menu Planning (ANSMP) 
which both require school menus to be planned using a USDA-approved computerized nutrient analysis 
program, offering schools more flexibility in planning menus as long as they met defined nutrient 
standards. (ANSMP relied on an outside resource to plan and conduct nutrient analysis on a school or 
SFA’s menus.) Finally, schools could elect a fifth option: “any reasonable approach” to planning menus as 
long as they met the defined nutrition standards and State Agency guidelines for the modified approach. 
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were measured in the context of substantial changes in the NSLP, and the levels of compliance with 
lunch standards may reflect implementation challenges unrelated to the CEP. The methods used by 
the study to assess meal quality emulated, to extent practical, the methods used by FNS and State 
Agencies to measure compliance with the meal component standards. However, the study simplified 
the approach to assessing schools’ compliance with meal component standards in order to reduce 
burden on participating schools. Therefore the research team did not have access to all of the 
information States routinely use for this purpose. While the study’s simplified methods allowed 
objective comparisons of CEP and non-CEP schools on meal components, the study results should 
not be interpreted as representing the same results FNS or State agencies would generate when they 
assess compliance with NSLP standards. Finally, readers are reminded that the sample was chosen to 
estimate the impact of the CEP and does not represent the broader population of LEAs in the six 
evaluation States, let alone the nation. 

8.1 Expectations about CEP Impacts on Meal Quality and Variety 

The CEP is intended to improve access to healthy school meals by providing free lunch and breakfast 
to all students in participating schools, and to reduce administrative burden for school foodservice 
operations. It was unclear at the start of the evaluation how the CEP might affect meal quality in 
participating schools versus those not participating. Hypotheses suggesting both positive and negative 
outcomes were plausible, as discussed below. 130 

On one hand, the CEP might increase meal quality and/or variety of foods offered to students by 
easing the administrative burden on school foodservice operators, allowing them to devote more time 
and resources to preparing higher quality meals. The CEP was expected to reduce administrative 
effort associated with household applications, the verification process, and tracking students based on 
meal eligibility status. In addition, participation in NSLP and SBP might increase under the CEP, 
potentially allowing participating schools to realize cost and production efficiencies. These changes 
might free up resources for higher quality meals and increase the variety of choices offered. 131 

Alternatively, the CEP might decrease meal quality and/or variety of foods offered for one of three 
reasons. First, changes in foods offered, particularly increased use of pre-prepared foods—which 
often have higher sodium content than similar school-prepared foods—could lead to higher overall 
sodium levels of meals offered. Second, increased NSLP and SBP participation in CEP schools might 
tax available school foodservice operator resources and lead to changes to streamline operations. 
Especially in those CEP schools experiencing increased SBP participation, one might reasonably 
expect some operational changes to accommodate such increases (e.g., serving breakfast in 
classrooms or grab-and-go style breakfasts) to minimize service time. Such a shift in meal service 
operations could, in turn, affect the variety or number of choices available to students on any given 
day. It may also limit the likelihood of serving certain types of breakfast foods to students due to 
service type limitations or clean-up concerns. For example, schools offering breakfast in the 
classroom may be challenged to offer hot foods if they lack the equipment or supplies required to 


130 For this evaluation, the measure of meal quality is based on how well schools met the relevant food 
component and nutrient specification standard for breakfasts and lunches offered during a specified week. 

131 In schools not previously offering SBP, of course, CEP participation will unambiguously increase the 
variety of SBP foods offered to students. 
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keep foods hot while delivering to classrooms. Third, if Federal revenues do not offset the loss of 
student payments for reimbursable meals and non-reimbursable foods, LEAs might need to reduce 
food costs in order to balance costs with revenues. Such cost-saving measures might reduce the 
quantities of more expensive foods such as whole-grain products. 

8.1.1 Research Questions Related to Meal Quality and Variety 

There are three exploratory research questions about the potential impact of the CEP on the 
nutritional quality of breakfasts and lunches served by schools: 

• Did the nutritional profile of school meals offered improve? 

• Did compliance with USDA nutrition standards improve, compared with similar LEAs that are 
not participating in the CEP? 

• Did the variety of food choices offered to students in school meals increase? 

The first two research questions deal with meal quality and the last question addresses meal choice 
and variety. 

The analysis uses a matched sample to compare outcomes in treatment versus comparison schools, 
controlling for differences in LEA- and school-level covariates, rather than examining changes over 
time associated with CEP participation. As discussed in Chapter 2, this posttest-only comparison 
design was the most feasible approach to estimate the difference in outcomes attributable to the 
implementation of the CEP in the treatment schools. 

To address the exploratory research questions, the evaluation collected information on reimbursable 
lunches and breakfasts offered using a self-administered Menu Survey in schools participating in the 
CEP and in non-participating schools in similar LEAs. The survey of cafeteria managers was 
designed to provide comprehensive information on the nutritional profile of meals and food choices 
these schools offered to students. The nutritional profiles of meals were compared to the then- 
applicable nutrition standards for school meals. The sections below describe the outcome measures 
and the data, as well as the methods used to address the research questions. 

8.2 Outcome Measures 

The Menu Survey instrument collected detailed information on reimbursable meals offered in a full 
week of lunches and breakfasts, which was used to construct the profile of meals offered in the study 
sample. (To be eligible for reimbursement, school meals must meet established nutrition standards as 
determined by averaging nutrients in meals offered over a school week, and by quantifying minimum 
daily and weekly food components for each grade grouping.) A USDA-approved nutrient analysis 
software package 132 was used by trained coders to conduct the nutrient and food component analysis 
of all menus. Further details about data processing are described in Appendix 8B. 

A weighted average approach to analysis was used for the nutrient specifications (calories, saturated 
fat and sodium levels), based on the planned number of servings and portion sizes of foods offered at 


132 NUTRIKIDS version 14.0.0 from Heartland School Solutions, which included the USDA Child Nutrition 
Database Release CN16. 
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each meal and the planned total number of meals, consistent with the methodology State agencies 
follow in assessing the nutrient specification compliance for NSLP and SBP meals offered under the 
new meal standards. A weighted analysis takes into account the number and types of foods planned 
for meals, and gives greater weight to foods offered more frequently to students, thereby emphasizing 
the relative nutrient contribution of foods offered, rather than weighting all foods equally within each 
meal component group. 

For the meal component analysis, an unweighted assessment of meal components offered for each 
school’s week of lunch menus was completed. The analysis used specifications to replicate as closely 
as possible the functionality of the certification tool and guidance provided by FNS on May 31, 2012 
to all State CN directors for measuring compliance with the new meal component requirements. 133 
(LEAs must be certified as meeting the meal component requirements in order to receive the 
additional 6 cents per meal reimbursement authorized by the F1F1FKA.) 

8.2.1 Measures Used for NSLP Meal Quality Assessment 

Depending on the meal component, the new meal pattern for lunch requires either a minimum number 
or a range of daily and weekly servings, as measured in either cups or ounce equivalents. To 
determine whether lunch menus met the new meal component requirements for meals offered, the 
servings offered in each food group (and subgroup) category were assessed based on combinations of 
food items and portion sizes offered in the various reimbursable meal choices reported in the Menu 
Survey for the target week for lunch. 134 The minimum quantity of daily and weekly amounts of food 
components and vegetable subgroups offered in reimbursable meals were compared with the 
prescribed minimums for a given grade/age grouping in lunch, in addition to the subgroup 
requirements as indicated in Exhibit 8.1 below. Details on grade-specific daily and weekly minimums 
in the Final Rule are included in Appendix 8A. 

In addition, compliance with key dietary specifications for lunches offered in all schools was assessed 
in each grade/age grouping, using the daily average from a five-day week for calories, and percent of 
calories from saturated fat. Although the Target 1 sodium standards were not yet in effect during the 
study’s data collection, these standards were used as a benchmark to describe the sodium content of 
breakfasts and lunches offered. 135 Due to concerns about respondent burden, the evaluation team did 


133 The Certification Tool, developed by FNS to assist SFAs and State agencies in measuring compliance with 
the new school nutrition standards, includes menu worksheets to assess compliance with daily and weekly 
meal pattern requirements. SFAs must complete the worksheet for one week of menus offered by grade 
group (K-5, 6-8, and 9-12), showing food components and quantities by each reimbursable meal offered or 
use a UDSA-approved tool provided by a software vendor, and submit the worksheet to their State agencies 
to be certified to receive the additional 6 cent lunch reimbursement. 

134 The study did not collect data on every possible combination of reimbursable meals offered at each meal 
(information routinely collected by State agencies); consequently, the evaluation team made assumptions 
about possible combinations based on available information from the menu survey, printed menus, and 
production records. 

135 The Final Rule established a 10-year phase-in period for sodium standards in school meals, with three 
sodium level targets by grade for breakfasts and lunches over the 10-year period, the first of which takes 
effect in SY 2014-15. The evaluation team, in consultation with FNS, used the Target 1 sodium standards 
for both breakfast and lunch comparisons. See Appendix 8A. 
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not collect the data (labels of pre-prepared foods used in meals) needed to assess the compliance with 
the trans fat requirement. 

Exhibit 8.1: Meal Component and Subcomponent Measures for NSLP Meals Offered 

• Fruit: daily and weekly minimum (cups) 

Fruit juice limit to no more than half of weekly total fruit 

• Vegetables: daily and weekly minimum (cups) 

Dark green vegetables (weekly cups) 

- Orange vegetables (weekly cups) 

Legumes (weekly cups) 

- Starchy vegetables (weekly cups) 

- Other vegetables (weekly cups) 

- Vegetable juice limit to no more than half of weekly total vegetable 

• Grains: daily and weekly minimum (ounce equivalents)* * 

Half the total weekly amount of grain must be whole grain-rich 

- Grain-based desserts limit to no more than 2 oz equivalents per week 

• Meats/meat alternates: daily and weekly minimum (ounce equivalents)* 

• Fluid milk: daily and weekly minimum (cups) 

Milk Types can include only fat free (flavored or unflavored) and unflavored low-fat milk 

• FNS Policy Memorandum SP 26-3013 (February 25, 2013) waived the weekly maximums on grains and 
meat/meat alternates through SY 2013-14 and considers SFAs compliant if weekly minimums of grains and 
meat/meat alternates are met. 

8.2.2 Measures Used for SBP Meal Quality Assessment 

During SY 2012-13, when Menu Survey data were collected, the new meal standards for the SBP 
were not yet in effect, except for the fat restrictions on fluid milk for meals offered. All but two LEAs 
(one CEP and one comparison LEA) reported using a traditional component-based menu planning 
approach (as indicated in response to the pre-visit telephone survey). The evaluation team, in 
consultation with FNS, therefore decided to use the traditional meal pattern requirements for the SBP 
comparison on meal standard compliance for all schools in the sample. 

The traditional meal pattern for school breakfast requires minimum component-specific quantities, 
and the same standards apply to all grades from K-12. Schools using Food-Based Menu Planning 
must offer food items in the specified minimum amounts. These are: 

• 1 serving of juice/fruit/vegetable, and 

• 1 serving of milk, and 

• A combination of grains/breads & meat/meat alternate components consisting of: 

- 2 servings of grains/breads, OR 

- 2 servings of meat/meat alternate, OR 

- 1 serving of meat/meat alternate and 1 serving of grains/breads, OR 

- An equivalent combination of meat/meat alternate and grains/breads 
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An “equivalent combination” means food items in total are credited as meeting 1 or more of required 
servings of these components. For example, if a breakfast pizza consists of 1 % servings of 
grains/breads and 3 A serving of cheese (a meat alternate), this combination food would be credited 
with 2 total servings of these components. (See Appendix 8 A for the Traditional Food-Based Meal 
Pattern for Breakfast, which describes the minimum amounts of food components and items for 
grades K-12.) In addition, the evaluation team assessed compliance with the daily milk types offered. 
Finally, the evaluation team measured the sodium levels of all breakfasts offered using the daily 
average from a week of menus, and compared those with the Target 1 standards for breakfasts. 136 

8.2.3 Measures Used for the Variety of Food Choices for NSLP and SBP 

Schools often provide students with a choice of entree and other meal components as part of the 
planned menu. To construct measures of variety of food choices offered, it is necessary to develop an 
appropriate classification of these offerings. Within a single planned meal, students may be offered 
choices of milk, fruit, vegetable, meat/meat alternate, or grain food components, with entrees often 
comprising more than one food component. For example, a hamburger on a bun includes both 
meat/meat alternate and grain food components, and students typically have no choice about 
accepting just one of those two components—the bun without the hamburger, or vice versa. It was 
therefore deemed inappropriate to implement a component-based variety measure for entrees. Instead, 
each entree was considered a single “choice,” even if comprised of multiple food components. 

To assess the variety of food choices offered for lunches, the evaluation team quantified the number 
of choices available to students in each school and meal within each food component category: milk, 
fruit, vegetables, grains (not part of an entree), entrees (including meat/meat alternates as well as 
meat/meat alternate combined with a grain and sometimes a vegetable), and desserts (if offered). 
Desserts included grain-based desserts such as cakes, cookies, cobblers, and pastries, as well as other 
desserts such as gelatin, pudding, sherbet or ice cream. Similarly, to assess the choice and variety of 
foods offered for breakfasts, the evaluation team quantified the number of choices available to 
students in each school and meal within each breakfast food component category: milk, 
fruit/vegetables/juice, meat/meat alternates (not part of a combination entree), bread/grains (not part 
of a combination entree), and combination entrees that included two or more food components (for 
example, a breakfast sandwich). A higher count of available choices for each meal was inteipreted as 
greater variety offered. 

8.3 Data and Methods 

Menu Survey data were collected from cafeteria managers in an elementary, middle, and high school 
in 52 LEAs, amounting to 81 treatment schools and 75 comparison schools (see Exhibit 8.2 for 
details). 137 In cafeterias identified as “shared” between two schools in the sample, the cafeteria 
manager completed one Menu Survey but the evaluation team coded data separately for each grade 
grouping to provide information on the portion sizes and number of planned servings for each 
respective grade grouping. Trained coders reviewed the Menu Survey data and systematically 
processed the data using the NUTRIKIDS menu planning software system. 


136 As above, sodium content was assessed using grade-specific Target 1 standards for SBP. 

137 54 LEAs, 27 treatment and 27 comparison, were initially selected. Two comparison LEAs were 
subsequently found to be ineligible for this portion of the evaluation. 
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Exhibit 8.2: Sample Design for Community Eligibility Option Evaluation 



Treatment 

Comparison 

Total 

For NSLP 

Matched LEA sample for Menu Survey 

27 

25 

52 

Schools for Menu Survey 

81 

75 

156 

Number of daily lunch menus 

401 

362 

763 

For SBP 

Matched LEA sample for Menu Survey 

27 

25 

52 

Schools for Menu Survey 

81 

75 

156 

Number of daily breakfast menus 

402 

362 

764 


Target week in most LEAs consisted of 5 days. In 17 schools for lunch and 16 schools for breakfast the target 
week consisted of 4 days. See Chapter 2 for more details on the matching process. 


To assess the impact of the CEP on the quality and variety of meals offered, the evaluation team used 
a posttest-only comparison group design, which compares outcomes for CEP and a matched 
comparison group of non-CEP schools while controlling for various school characteristics. Outcomes 
were measured only after the CEP was implemented. The analytic model used was similar to the one 
used to generate findings discussed in Chapters 6 and 7. 

As in the other school-level analyses, matching occurred at the LEA level, and so the school samples 
were not balanced. Appendix 8C presents descriptive statistics on the characteristics of the LEA and 
school samples for the Menu Survey data collection. The results in this chapter use regression- 
adjusted treatment group means to control for the significant differences in school characteristics, 
including the percentage of students approved for FRP meals in the school, indicators for charter 
schools and Title I eligibility, and the percentage of students who are Black in the school. Appendix 
8C presents the details of the regression analysis that supports the impact estimates. Elnlike the data 
from administrative sources and for Components 1 and 2, the treatment group data for Component 3 
came only from LEAs that took up the CEP in all schools. 118 Therefore, the impact estimates should 
be interpreted as the treatment effect for LEAs in which all the schools participated in the CEP. 

Appendix 8C reports descriptive statistics related to meal quality and variety, and presents 
statistically significant differences between the treatment and comparison schools, according to t- 
tests. It also presents statistics for more detailed meal component subgroups and other standards for 
which the meal analysis software provided results. Descriptive results should be viewed with caution 
because of the differences in school characteristics, which may affect comparisons. 

8.4 Meal Quality and Choice Results 

This section presents exploratory findings on the impact of the CEP on meal quality (described in 
terms of meeting food component and nutrient standards) and the variety of menu choices, based on 
comparisons of these outcomes between CEP and non-CEP schools. The outcomes for meal quality 
indicate whether schools met the then-applicable food component and nutrient standards for lunch 
and breakfast (as of Winter/Spring 2013) and also the sodium targets that will be effective in SY 
2014-15. The outcomes for menu choice relate to the variety of food options within specific 


138 Component 3 included data collection on two domains: meal quality/choice and errors in counting and 
claiming reimbursable meals. 
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component categories available to students during lunch and breakfast. Both analyses rely upon the 
school level posttest-only comparison group design described above. 

8.4.1 Meal Quality 

A school met the requirement for a lunch food component if it met the grade-specific criteria for daily 
minimum for each food component, in each of the days for which menu data were collected, and the 
weekly minimum for that food component. The other subcomponent requirements were considered 
separately (see Appendix 8C for results, and results of daily and weekly minimums reported 
separately). For nutrient specifications of calories and percent of calories from saturated fat, a school 
met the requirement if the daily average (over all days within the week for which data were collected) 
met the grade-specific standard for that nutrient in a weighted analysis. The Target 1 sodium 
comparison was also included in this analysis. 

Exhibit 8.3 shows the regression-adjusted differences between treatment and comparison schools in 
the food components for lunches offered on three, four, or all five days of the week. There were no 
significant differences between treatment and comparison schools with the specified minimums for 
most food components in lunches offered, including fruits, grains, milk and meat, whether the 
measure was for three, four, or five days, lj9 although treatment schools were significantly more likely 
to offer vegetables (79 versus 57 percent). To meet the requirement for food components offered, 
schools had to meet the minimums both for all five days and the weekly component minimum. 140 

Appendix 8C provides additional detailed information on meal quality in treatment and comparison 
schools, by presenting results separately for daily and weekly meal components. There were no 
significant differences between treatment and comparison schools in the proportion meeting these 
minimums. 


139 As Exhibit 8.3 indicates, the difference in the percentage of schools meeting the grains standard for all five 
days approached significance, with the treatment group having the higher estimate. 

140 Only four days of menus were reported for a some schools due to scheduling, school holidays, or teacher 
conference days during the Target Week (15 schools had four days of breakfast menus, and 17 schools had 
four days of lunch menus. Schools that met standards for all four reported days and the prorated weekly 
standard were characterized as having met the standard for the week. 
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Exhibit 8.3: Differences between Treatment and Comparison Schools: Food Component 
Minimums for 3, 4 or all 5 Days of the Week for Lunches Offered 



Means for comparison group are unadjusted (raw means). Means for treatment group are regression-adjusted 
to control for differences between treatment and comparison schools. The results from the regression, including 
all covariates, appear in Appendix 8C. 

Meeting Requirements for a component means that the school met the daily minimum requirements for the 
school’s grade range on all days and also met the weekly minimum requirement. Meeting “All Requirements” 
means the school met all daily and weekly minimums for all meal components combined. 

***=p<0.01, **=p<0.05, *=p<0.1. 

Source: Menu Survey. 

Exhibit 8.4 provides the regression-adjusted percentage of treatment schools and the average 
percentage of comparison schools meeting the nutrient specifications and the future Target 1 for 
sodium for the lunches offered. There were no statistically significant differences between treatment 
and comparison schools for meeting nutrient specifications for calories and saturated fat. About four 
out of five schools in both groups came within 10 percent of the specifications for average calories. 
Appendix 8C (Exhibit 8C.4) provides details on the percent of schools that fell below or above the 
target calorie ranges for age/grade grouping. Nearly all treatment and comparison schools met or 
came within 10 percent of meeting the saturated fat standard for lunches offered. Elowever, treatment 
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schools were significantly less likely than comparison schools to meet the Target 1 sodium standard 
(which will not be in effect until SY 2014-15). Sixty-one percent of comparison schools met the 
future sodium target, as compared to 44 percent of treatment schools. 


Exhibit 8.4: Differences between Treatment and Comparison Schools: Nutrient Specifications 
and Sodium Target 1 for Lunches Offered 
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Means for comparison group are unadjusted (raw means). Means for treatment group are regression-adjusted to 
control for differences between treatment and comparison schools. The results from the regression, including all 
covariates, appear in Appendix 8C. 

The first of two intermediate sodium targets was used in the analysis (Target 1) for comparison purposes and will 
be in effect in SY 2014-15 but was not in effect when the Menu Survey data were collected. 

Meeting Requirements for a nutrient means that the school met the average daily requirements, based on the 
weekly menu for the school’s grade range. 

***=p<0.01, **=p<0.05, *=p<0.1. 

Source: Menu Survey. 

The same methods were used to estimate the CEP’s impact on meal quality for breakfasts offered. 
Exhibit 8.5 shows the regression-adjusted differences on the specified food component requirements 
for the treatment and comparison schools for breakfasts offered. Differences between treatment and 
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comparison groups were not statistically significant for the percentages meeting the food component 
requirements for juice/fruit/vegetable, milk, and the meat/grain combination, as well as all three food 
components combined. Compliance with requirements for juice/fruit/vegetables and fluid milk was 
universal or nearly so in both groups and only slightly less (92 percent) for the meat/grain standard 
among treatment schools. 


Exhibit 8.5: Differences between Treatment and Comparison Schools: Food Component 
Requirements for Breakfasts Offered 
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Means for comparison group are unadjusted (raw means). Means for treatment group are regression-adjusted to 
control for differences between treatment and comparison schools. The results from the regression, including all 
covariates, appear in Appendix 8C. 

***=p<0.01, **=p<0.05, *=p<0.1. 

Meeting Requirements means that the components meet the daily minimum quantities specified for the 
respective grade range on all days. Meets All Three Components requirement means schools met all meal 
component minimums combined. 

Source: Menu Survey. 

Unlike the analysis for lunches offered, the difference in the percentage of schools meeting the future 
Target 1 sodium standard for breakfast was not significant (see Exhibit 8.6). Seventy-six percent of 
comparison and 70 percent of treatment schools met or came within 10 percent of meeting the future 
Target 1 sodium standard. 
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Exhibit 8.6: Differences between Treatment and Comparison Schools: Future Sodium 1 Target 
for Breakfasts Offered 
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Means for comparison group are unadjusted (raw means). Means for treatment group are regression-adjusted to 
control for differences between treatment and comparison schools. The results from the regression, including all 
covariates appear in Appendix 8C. 

The Target 1 sodium standard was used in the analysis for comparison purposes and will be in effect in 
SY 2014-15, but was not in effect when the Menu Survey data were collected. 

***=p<0.01, **=p<0.05, *=p<0.1. 

# - varies by grade. 

Source: Menu Survey. 

8.4.2 Menu Choice and Variety 

The evaluation team also explored whether the CEP induced changes in the variety of food choices 
that schools offered to students in lunches and breakfasts, to address the final research question about 
meal quality and variety. Exhibit 8.7 provides the regression-adjusted number of food group choices 
offered for lunches for the treatment and comparison schools. The CEP did not have a statistically 
significant impact on choices offered for fruits, vegetables, grains, milk, and entrees. The estimated 
difference between treatment and comparison schools was less than 0.3 items for these categories, 
except for the difference of about 0.4 in the number of entrees offered. The only significant effect was 
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the difference of 0.09 in the average number of desserts offered. 141 Given the small number of 
desserts offered at both treatment and comparison schools, the impact on the number of desserts did 
not substantially alter the overall pattern that food choices at lunch were not different under the CEP. 


Exhibit 8.7: Impact of CEP on Average Number of Choices in Food Groups Offered at Lunch 



Comparison 

Treatment 

Impact 

Percent Impact 

Number of fruits and 100% juices 

2.58 

2.36 

-0.22 

-8.36% 

Number of vegetables 

2.32 

2.41 

0.09 

3.72% 

Number of separate 
grains/breads 

0.52 

0.66 

0.14 

26.15% 

Number of types of milks 

2.96 

2.84 

-0.12 

-4.2% 

Number of entrees 

3.72 

3.34 

-0.38 

-10.15% 

Number of desserts 

0.07 

0.16 

0.09** 

117.93% 


Means for comparison group are unadjusted (raw means). Means for treatment group are regression-adjusted to 
control for differences between treatment and comparison schools. The results from the regression, including all 
covariate results, appear in Appendix 8C. 

Entrees for lunches include meat/meat alternates and combination entrees (meat/meat alternates combined with 
grains, vegetables, or both). 

***=p<0.01, **=p<0.05, *=p<0.1. 

Source: Menu Survey. 

Exhibit 8.8 provides the regression-adjusted number of food group choices offered for breakfasts for 
the treatment and comparison schools. There was no statistically significant impact of CEP 
participation on choices offered for grains, meat, milk, and entrees. The only suggestive evidence of 
an effect (at the 10 percent significance level) was on the number of fruits and vegetables offered. 

The number of choices of fruits/vegetables offered at treatment schools was approximately three, and 
at comparison schools was approximately three and a half. The difference of half an extra choice is 
the estimated impact of the CEP, which represents a 15 percent decrease in number of choices 
available in relative terms, but this finding is only suggestive and not significant at the standard 5 
percent level. 


141 At most schools, only one type of dessert is offered for a given meal, so the mean number of desserts 
reflects the percentage of schools offering dessert (see Appendix 8C for percentage of schools offering 
dessert). 
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Exhibit 8.8: Impact of CEP on Choices Offered at Breakfast 





Percent 


Comparison 

Treatment 

Impact 

Impact 

Number of fruits/vegetables/100% juices 

3.44 

2.91 

-0.53* 

-15.4% 

Number of separate grains/breads 

5.92 

5.27 

-0.64 

-10.9% 

Number of separate meats/meat alternates 

0.92 

0.75 

-0.17 

-18.2% 

Number of types of milks 

2.63 

2.48 

-0.15 

-5.7% 

Number of combination entrees 

0.78 

0.61 

-0.17 

-21.5% 


Means for comparison group are unadjusted (raw means). Means for treatment group are regression-adjusted to 
control for differences between treatment and comparison schools. The results from the regression, including all 
covariates results appear in Appendix 8C. 

Combination entrees for breakfasts include only entrees composed of a meat/meat alternate and at least one 
other meal component (typically a grain). Counts of meat/meat alternates and grains that were not part of a multi- 
component item were computed separately. 

***=p<0.01, **=p<0.05, *=p<0.1. 

Source: Menu Survey. 

8.4.3 Discussion 

Overall, the analysis indicates few significant differences in meal quality indicators and in food 
choices between the treatment and comparison schools. These are exploratory findings suggestive of 
impacts. Were the analysis to control for the multiple hypothesis tests being conducted, the 
conclusion might be that no between-group differences exist. 142 A further caution is that, given the 
observed differences between the samples, it is also likely that the groups are different on unobserved 
characteristics for which the available covariates do not control, and these unobserved characteristics 
might have contributed to the differences in outcomes. 

Meal Quality 

There were no statistically significant differences between treatment and comparison schools in 
meeting the food component minimums for fruits, grains, meat/meat alternates or milk for lunches 
offered. The slight increase in the percentage of schools meeting the vegetable component minimum 
among the treatment schools is consistent with the results of the administrative cost interview 
( described in Chapter 3), which indicated that, among the one-third of the foodservice directors in 
treatment schools reporting that the quality of foods increased, the addition of fruits and/or vegetables 
was the most co mm on specific change reported. 

Similarly, there were no statistically significant differences between the treatment and comparison 
schools for meeting the nutrient specifications for the calorie range or the percent of calories from 
saturated fat. However, comparison schools were significantly more likely to meet the relevant first 
intermediate sodium target (Target 1) than the treatment schools. As stated previously, schools were 
not expected to meet Target 1 until SY 2014-15, and so this analysis simply provides context about 
the actual sodium values reported relative to the future target. 


142 As discussed in Chapter 2, the evaluation specified confirmatory outcomes in advance and treated all other 
outcomes as exploratory, in order to maximize the power to detect impacts on the most important 
outcomes. Adjustment for multiple hypothesis tests would substantially reduce the power to detect any 
impacts. 
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The evaluation provides some evidence suggesting a possible explanation for higher sodium values 
for lunches in treatment schools: that the schools may have made changes to offerings that allowed 
them to serve lunch more quickly. Evidence for this explanation comes from the one-third of CEP 
foodservice directors that reported menu changes attributable to the CEP in the Administrative Cost 
Interview. An increased reliance on pre-prepared or processed foods could lead to an increase in 
sodium, which is typically higher in processed foods than those prepared from scratch. However, the 
evaluation did not specifically classify foods to allow an analysis of food sources of sodium, so this is 
only one plausible explanation for the result. 

For breakfasts offered, the meal quality analysis found no significant differences between treatment 
and comparison schools in the food component requirements for juice/fruit/vegetable, milk, and the 
meat/grain combination. Unlike the results seen in lunches offered, the comparison of schools 
meeting the future sodium target (Target 1) revealed no statistically significant difference. 

As noted above, the methods used by the study to assess meal quality emulated the methods used by 
FNS and State Agencies to measure compliance with the meal pattern and nutrient standards, 
although the study did not have access to the same information available to FNS and the States. 
Consequently, the study’s approach likely resulted in a more conservative assessment of school 
meals. See Appendix 8B for discussion of this aspect of the methods. 

Choice and Variety Offered 

For lunches offered, there were no statistically significant differences in the number of choices 
offered for fruits, vegetables, grain, milk and entrees between the treatment and comparison schools. 
These results appear to be consistent with the responses of the TEA foodservice directors surveyed in 
the Administrative Cost Interview, described in Chapter 3, which indicated that the majority reported 
no change to the variety of foods offered as a result of the CEP. The only significant impact of the 
CEP was the evident increase in the number of desserts offered. While the new meal pattern for 
lunches does allow for desserts to be offered, they must be accounted for in the overall dietary 
specifications (calories, saturated fat, trans fat and sodium) (USDA, 2013d), and meal pattern 
requirements limit grain-based desserts to no more than two oz per week, providing an inherent 
constraint to including them with lunches. 143 It is also important to note that given the small number 
of desserts offered at both treatment and comparison schools, the impact on the number of desserts 
did not substantially alter the overall pattern that food choices at lunch were not different under the 
CEP. 

For breakfasts offered, the results suggest that the CEP may have reduced the number of choices of 
fruits, vegetables, and juices by an estimated one-half item. This finding, and the consistently 
negative (though not statistically significant) impact estimates for other choices are consistent with 
self-reports from Administrative Cost Interview data. As discussed in Chapter 3, the majority of LEA 
foodservice directors in treatment LEAs reported no change to the overall variety of foods offered as 
a result of CEP participation. However, about one-third reported changes to the types of food served 
to allow for quicker meal service. Offering grab-and-go food was the most common change reported. 


143 FNS Guidance indicates that a school may offer a total of 2 oz equivalent or less of grain-based desserts 
each week, but the distribution across the week can vary. For example, a 2 ounce equivalent dessert may be 
offered once per week, or a 0.5 oz equivalent dessert may be offered four times per week (USDA, 2013c). 
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This operational change, or others such as breakfast in the classroom in CEP schools, may help to 
explain why the data reflect slightly fewer choices of some types of foods offered in CEP schools. 

The results about meal quality and choice suggest few meaningful differences in outcomes for CEP 
and comparison school menus. For the levels of meal quality and choice in both groups, the timing of 
the data collection relative to the shift in applicable nutritional standards may well be an explanatory 
factor. The new NSLP standards were in their first year of operation. 

From a broader perspective, the findings from this assessment about levels of meal quality and variety 
are comparable to past national studies that have assessed the nutrient content of the NSLP and SBP, 
which indicate both progress and challenges in meeting USDA standards. Since the early 1990s, FNS 
has sponsored the School Nutrition Dietary Assessment (SNDA) studies to provide information on 
key characteristics of the school meal programs and the food and nutrient content of school meals. 
Most recently, the fourth SNDA study (SNDA-IV) collected data from national samples of school 
districts and schools in school year (SY) 2009-10. For the SNDA studies, the nutrient content of the 
average meals offered and served in the nation’s schools was compared with standards in effect at the 
time—the School Meals Initiative (SMI) nutrition standards—as well as several recommendations 
from the Dietaiy Guidelines for Americans. Past SNDA studies documented progress toward meeting 
the goals for nutrients in NSLP lunches, but found that a considerable percentage of schools 
continued to offer lunches that did not meet SMI standards, particularly for fat, saturated fat, and 
calories. (Fox et al., 2012) From the most recent study, SNDA-IV, few schools offered average NSLP 
lunches that met all of the SMI standards; only 14 percent of schools offered NSLP lunches that met 
all of the SMI standards Similarly, SNDA-IV also reported that only 15 percent of schools offered 
average SBP breakfasts that met all of the SMI standards. 
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9. Conclusions 


Data collected for this report provide a broad perspective on the implementation and impacts of the 
Community Eligibility Provision (CEP). This chapter summarizes the main findings of the 
Implementation and Impact Studies of the CEP. 

9.1 Implementation Study Results 

The Implementation Study drew on four main data sources. The first seven CEP States provided data 
from administrative records on participating, eligible non-participating, and near-eligible LEAs and 
schools. State Child Nutrition Officials participated in three rounds of telephone conversations in 
2012 and one in 2013 about their experiences with the CEP. LEA foodservice directors in the first 
seven CEP States provided data on their experiences and perceptions regarding the CEP via the 
implementation web surveys. These data reflect the specific experiences of participating States and 
LEAs, and the process and issues could well have been different elsewhere. Reaching beyond the first 
seven CEP States, telephone interviews with Title I directors in all 51 State Education Agencies 
provided insights into the use of FRP meals eligibility data for educational programs, and how these 
programs might be affected by the elimination of FRP meals applications under the CEP. 

Three States—Illinois, Kentucky, and Michigan—and their LEAs began implementing the CEP in SY 

2011- 12, and continued operating the CEP in SY 2012-13. Four additional States, including the 
District of Columbia, New York, Ohio, and West Virginia, and their LEAs implemented the CEP in 
SY 2012-13. Four more States—Florida, Georgia, Maryland, and Massachusetts—were well- 
positioned to implement during SY 2013-14 despite numerous challenges. The first seven 
implementing States identified, notified and recruited LEAs to participate. In turn, the LEAs learned 
about the CEP, made decisions about whether to implement and in which schools, and communicated 
with schools and the community. Some States and LEAs had already noticed increased meal 
participation, and according to States, LEAs were excited about the CEP. According to the first seven 
States, key challenges at the State level were (1) the limited time during initial implementation to 
understand the CEP, make decisions about participation, and implement it, and (2) understanding and 
addressing the implications of the CEP for educational programs that use individual student FRP 
meals certification data. At the LEA level, the most frequently reported barriers included financial 
concerns and uncertainty about the impacts of the CEP on NSLP and SBP participation. Equity issues 
and operational challenges (especially for LEAs with both eligible and non-eligible schools) were 
also important concerns. Addressing the timing and information issues for States is a relatively 
straightforward problem. However, telephone interviews with all 51 State Title I directors indicated 
that States routinely use FRP data for multiple education-related purposes, so the lack of such data 
under the CEP represents a widespread challenge. 

Across the first seven States, a total of 420 LEAs and 2,312 schools participated in the CEP in SY 

2012- 13. On average, 32 percent of eligible regular and charter LEAs, and 29 percent of eligible 
schools, participated in the CEP. Participation rates for eligible regular and charter LEAs ranged 
considerably—from 24 percent in Michigan to 65 percent in West Virginia. Based on the number of 
near-eligible LEAs, it appears that the potential to increase the number of eligible LEAs was small in 
most States; only Illinois and Ohio had more than 50 near-eligible LEAs. 
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Among participating LEAs eligible to participate LEA-wide, 92 percent offered the CEP at all 
schools, and 97 percent of schools offered the CEP, on average. On the other hand, among 
participating LEAs that were not eligible LEA-wide, 53 percent offered the CEP at all eligible 
schools, and on average, 79 percent of eligible schools offered the CEP. 

Compared with eligible non-participating regular and charter LEAs, participating LEAs of these types 
had more students, on average, higher ISPs and FRP meals eligibility percentages, higher percentages 
of students in grades K-5, and higher percentages of students who are Black. Despite their larger 
average size, participating regular and charter LEAs were more often very small (with enrol lm ents 
under 500 students); they also were more often urban and more often charter schools. These patterns 
were replicated in most (although not all) of the first seven States. 

Four factors were significantly associated with CEP participation by LEAs: ISP, enrollment, State, 
and charter status. The odds of participation rose dramatically with each additional 10 percentage 
points in the ISP, especially between 40 and 70 percent. The odds of participation also increased for 
LEAs with larger numbers of schools. However, LEAs with below-average-size schools were more 
likely to participate, possibly because of differences in school-level eligibility. Although charter 
LEAs were more likely to participate than regular public LEAs, this finding was significant only in 
Ohio. The above relationships were observed after controlling for the other factors in the analytic 
model, so they may differ from the descriptive results (such as the participation rates by State 
computed without controlling for differences in LEA demographics). Evidence from State CN 
interviews suggests that the differences by State and charter status may reflect how the CEP was 
implemented; differences in school-level eligibility may also have contributed to variation in 
participation across LEA types. 

9.2 Impact Study Results 

As discussed in Chapter 2, the Impact Study used a matched comparison design, and impact estimates 
were computed from differences in outcomes between the treatment group (participating LEAs and 
schools) and the comparison group (non-participating LEAs selected to be similar in likelihood of 
CEP take-up, and selected schools in these LEAs). Outcome data were drawn from State 
administrative records, the PEAR survey, in-person interviews and other on-site data collection, with 
the largest samples devoted to the participation and revenue impacts. 

While the impact estimates control statistically for known differences between the groups, there are 
some caveats to the findings, and the results must be inteipreted with appropriate caution. These 
estimates, at best, measure the effect of the CEP on the specific group of LEAs that were observed: 
those that chose to participate in the first two years of availability in the Year 1 States (Illinois, 
Kentucky and Michigan) and those that chose to participate in the first year of availability in the three 
Year 2 States (New York, Ohio, and West Virginia). (The District of Columbia was excluded because 
the matched comparison design could not be used there.) These estimates generalize neither to LEAs 
in these States that are not comparable to the participating LEAs, nor to LEAs in other States. The 
matching and statistical controls do not account for potentially important, unmeasurable 
characteristics that determine the take-up decisions of LEAs, such as the perception that the LEA will 
benefit from adopting the CEP. Nevertheless, the study used the most rigorous and feasible methods 
to separate the impact of the CEP from the other factors that could have affected the outcomes of 
interest. The results for impacts on participation and revenues for NSLP and SBP are based on the 
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strongest component of the evaluation, which uses a time-series of pre- and post-implementation data 
to strengthen the controls for differences in relevant characteristics between the treatment and control 
groups. Other impact results are based on a posttest-only comparison design and should be considered 
exploratory. 

Judged by the confirmatory outcome selected for the evaluation, the CEP was clearly successful: it 
significantly increased student participation in NSLP and SBP, and Federal reimbursements per 
reimbursable meal for these programs were significantly greater among the participating LEAs. The 
CEP’s impact on student participation in school meal programs was surprisingly consistent across 
programs, increasing participation by 5 to 9 percent (relative to the lunch and breakfast programs’ 
participation in the matched comparison schools, after controlling for differences between the two 
groups). The CEP’s impact on Federal reimbursements per meal was a significant increase of about 6 
percent for the NSLP and 2 percent for the SBP. Total Federal revenue per student increased by 13.5 
percent. On the other hand, the CEP did not appear to have an impact on any of type of non-Federal 
revenue. Taken together, the finding of increased Federal funding and the lack of evidence that other 
revenues significantly declined suggest that the CEP did not have an adverse overall effect on LEA 
foodservice revenues, and may have produced a net gain for participating LEAs. 

The CEP requires LEAs to offer the SBP, and it was available in all schools in nearly all participating 
LEAs and non-participating LEAs in the comparison group. Study data did not indicate that the CEP 
had an impact on availability of the SBP. After controlling for differences between groups, 
participating LEAs used traditional line service less often at breakfast and served breakfast in the 
classroom more often, as is recommended to increase SBP participation (USDA, 2013e) but 
complicated to implement under conventional meal counting and claiming procedures. Participating 
LEAs were less likely to report that they offered a choice of foods at breakfast. The reduction in 
choice of foods may be li nk ed to the greater use of in-classroom breakfast. 

As expected, CEP participation appeared to reduce time spent by school and LEA staff on distributing 
and processing applications for free or reduced-price meals, verifying income of free/reduced price 
students, and meal payment collections and accounting. The combined savings for these activities 
were 68 minutes per student per year, representing labor cost savings of about $29 per student per 
year. The CEP had no impact on the staff time and costs associated with direct certification. On the 
other hand, the CEP appeared to increase the staff time and cost of counting and claiming 
reimbursable meals from less than 30 minutes per student per year to almost an hour, offsetting half 
of the savings in other activities. Possible reasons for this result include: increased participation in 
school meal programs, and new (and sometimes more manual) meal counting and claiming 
procedures. It is too early to tell whether the increased time for meal counting and claiming reflects 
an early implementation challenge, and adaptation to the new procedures and planned automation by 
States might reduce this impact over time. On the other hand, the CEP could produce net savings in 
meal counting and claiming costs in the long run, if schools could reduce the complexity and cost of 
automated systems for meal counting and payments. Treatment and comparison schools did not differ 
in the total time and cost spent on administrative tasks overall or for any category of worker. 
Descriptive data from interviews and surveys suggest that, for the majority of LEAs, the CEP did not 
require LEAs to increase staff, the time spent on meal counting and claims was not a concern, and the 
CEP made it easier for LEAs to break even or at least had no impact on the foodservice bottom line. 
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The CEP reduced the overall rate of certification errors and had little or no impact on errors in 
counting meals (at the cashier level) and claiming meals for reimbursement. The CEP did not appear 
to affect errors in direct certification, which are rare. The CEP eliminated application processing 
errors that, in the comparison schools, resulted in approximately 6.6 percent of applicants having the 
wrong certification level (too high or too low). The CEP also eliminated procedural errors in 
application processing that, in the comparison schools, affected 20.2 percent of approved applicants 
and 16.9 percent of denied applications. Procedural errors (such as missing certification documents 
or incomplete applications) do not necessarily result in certification errors but indicate a lack of 
integrity in the process. The CEP had no significant impact on cashier error in identifying 
reimbursable meals, despite concerns that increased participation might lead to more error by 
speeding up service lines. One measure of meal claiming error was significantly higher in 
participating schools, but there was no impact on overall meal claiming error. The primary reason for 
errors in claims submitted for CEP schools to the State was that the LEA used the wrong claiming 
percentage, and usually the claiming percentage used was lower than the LEA was entitled to use. 
This problem might be readily addressed through training or through claims processing by the State 
(which may have corrected some or all of the detected errors in claiming percentages). The baseline 
levels of claiming error in the NSLP and SBP appear to be quite small, at least in the sample, so the 
lack of a positive impact from the CEP is not a reason for concern. 

For a broad range of meal quality outcomes, there was no evidence that the CEP had a significant 
impact. At lunch, CEP schools tended to offer more vegetables, but were less likely than comparison 
schools to have met the (soon-to-be-required) Target 1 sodium standard. There was no evidence of 
impacts on meeting NSLP requirements for other food components, the calorie range requirement, or 
the percent of calories from saturated fat. There were no significant differences in meeting food 
component requirements or the Target 1 sodium standard for breakfasts offered. 

The CEP appeared to have little if any impact on the number of choices offered in meal components. 
There were no significant differences in the number of choices offered for fruits, vegetables, grain, 
milk and entrees for lunches between the treatment and comparison schools, nor was there a 
significant difference in choices for breakfast components.For lunch, the only significant impact of 
the CEP was a small increase in the number of desserts offered, although the number of desserts 
offered was minimal and well within allowed limits for both CEP and comparison schools. Changes 
in the types of food offered to serve more quickly, and shifts to breakfast in the classroom in CEP 
schools were reported but had no significant effect onthe number of breakfast choices offered in CEP 
schools. 

Summing up across all of the evaluation results, the implementation of the CEP in its first two years 
was successful in perhaps the two most important dimensions. First, the take-up of the CEP was 
widespread among eligible LEAs, despite the uncertainties about impacts on finances and operations; 
further, participating LEAs were both well-satisfied and likely to continue using the CEP. Second, the 
CEP appeared to increase NSLP and SBP participation and the associated Federal reimbursements, 
while not adversely affecting the financial bottom-line of foodservice operations. The CEP eliminated 
substantial levels of error in the processing of FRP meals applications, as well as eliminating the costs 
of processing and verifying these applications. For most other outcomes, there were little or no 
impacts, either positive or negative. 
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The results point to several key challenges for the future of the CEP. First, there is substantial 
skepticism about the feasibility of the CEP among eligible non-participating LEAs; wider adoption of 
the CEP may require some efforts to address this skepticism. Second, the need for household income 
data for Federal and State educational programs will continue to pose perhaps the greatest 
implementation challenge. Third, impact findings suggest that the LEAs experience new challenges in 
meal claiming under the CEP, at least during initial implementation, based on the evidence of 
increased administrative costs and errors. Several currently-implementing CEP States have already 
implemented automated systems to prevent errors in the use of claiming percentages, and others 
indicate that they plan to do so. As this practice becomes more widespread, it would presumably 
eliminate the main source of claiming error observed in CEP schools. Finally, the findings on meal 
quality and choices provide evidence of little change in meal quality, suggesting that the participating 
LEAs have not reduced meal quality to contain costs but also that any resources freed up by the CEP 
have not led to broad imporvements in meals. The only significant effects on meal quality were the 
positive impact on vegetables offered and the negative impact on meeting the future intermediate 
sodium target. Given the timing of the study, it is too early to determine whether the observed effects 
on meal quality are merely transient (reflecting CEP implementation, introduction of new standards, 
or both) or likely to be sustained. Across the multiple outcomes assessed by the Impact Study, the 
clear and positive impacts on NSLP and SBP participation and reimbursements represent the most 
notable results of implementing the CEP. 


Abt Associates 


9. Concusions | pg. 167 





Community Eligibility Provision Evaluation: Final Report 


References 


Benjamini, Yoav, and Hochberg, Yosef. (1995). Controlling the False Discovery Rate: A Practical 
and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society, Series B 57(1): 
289-300. 

Dunn, O. J. (1961). Multiple Comparisons Among Means. Journal of the American Statistical 
Association 56(293): 52-64. 

Federal Register. (2012). Nutrition Standards in the National School Lunch and School Breakfast 
Programs; Final Rule. Retrieved from http://www.gpo.gov/fdsys/pkg/FR-2012-01-26/html/2012- 
1010.htm 

Food Research and Action Council (FRAC). (2012). School Meals Legislation by State. Retrieved 
from http://www.frac.org/pdf/school meals legislation.pdU 

Fox, May Kay, et al. (2012). School Nutrition Dietary> Assessment Study IV, Vol. I: School 
Foodservice Operations, School Environments, and Meals Offered and Sensed. Alexandria, VA: U.S. 
Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis. 

Institute of Medicine. (2010). School Meals: Building Blocks for Healthy Children. Washington, DC: 
The National Academies Press. 

Moore, Quinn, Conway, Kevin, and Kyler, Brandon. (2012). Direct Certification in the National 
Lunch Program: State Implementation Progress School Year 2011—2012. Alexandria, VA: U.S. 
Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis. 

National Research Council. (2012). Using American Community Survey Data to Expand Access to the 
School Meals Programs, Report of the Panel on Estimating Children Eligible for School Nutrition 
Programs Using the American Community’ Survey, A.L. Schirm and N.J. Kirkendall, Editors. 
Committee on National Statistics, Division of Behavioral and Social Sciences and Education. 
Washington, DC: The National Academies Press. 

Ponza, Michel, Gleason, Philip, Fluley, Lara, and Moore, Quinn. (2007). NSLP/SBP Access, 
Participation, Eligibility>, and Certification Study—Erroneous Payments in the NSLP and SBP, 

Volume I: Study Findings. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition 
Service, Office of Research, Nutrition and Analysis. 

U.S. Department of Agriculture, Food and Nutrition Service. (2013a). National School Lunch 
Program: Participation and Lunches Served. Retrieved from 
http://www.fns.usda.gov/pd/slsummar.htm 

U.S. Department of Agriculture, Food and Nutrition Service. (2013b). School Breakfast Program 
Participation and Meals Served. Retrieved from http://www.fns.usda.gov/pd/sbsummar.htm 

U.S. Department of Agriculture, Food and Nutrition Service. (2013c). Final Rule: Nutrition Standards 
in the National School Lunch and School Breakfast Programs. Questions & An s wers for Program 
Operators. Retrieved from http://www.fiis.usda.gov/cnd/govemance/Policy-Memos/2012/SP10- 
2012av7.pdf 


Abt Associates 


References | pg. 169 











Community Eligibility Provision Evaluation: Final Report 


U.S. Department of Agriculture, Food and Nutrition Service. (2013d). Provisions 1, 2, & 3 Fact 
Sheet. Retrieved from http://www.fhs.usda.gov/cnd/govemance/prov-l-2- 
3/Provl 2 3 FactSheet.htm . 

U.S. Department of Agriculture, Food and Nutrition Service. (2013e). Breakfast in the Classroom 
Worksheet. Retrieved from http://www.fns.usda.gov/sites/default/files/toolkit_classroom.pdf 

U. S. Department of Education. (2011). Improving Basic Programs Operated by Local Educational 
Agencies (Title I, Part A). Retrieved from http://www2.ed.gov/programs/titleiparta/index.html 


pg. 170 | References 


Abt Associates 







Community Eligibility Provision Evaluation: Final Report 


Glossary: Community Eligibility Provision Evaluation 


Average Daily Attendance 

A figure representing a school or school district’s attendance per day averaged over a period of time. 

Bi-Weekly 

Every two weeks. 

Central LEA Foodservice Staff 

Foodservice staff working in the administrative office of the school foodservice department. These 
may include the foodservice director, supervisors, bookkeeper and possibly, the business manager. 

CEP Schools 

Schools participating in the Community Eligibility Provision, formerly known as the Community 
Eligibility Option (CEO). 

Claiming Percentage 

The percentage of meals reimbursed with Federal funds at the free meals rate for schools participating 
in the Community Eligibility Provision (CEP). 

Claims for Reimbursement 

A claim submitted to the State Agency on a monthly basis for reimbursement for meals served under 
the National School Lunch and School Breakfast Programs. 

Common Core of Data (CCD) 

The Common Core of Data compiled by the National Center for Educational Statistics, is a 
longitudinal database of key characteristics of LEAs and schools. Most of the data are obtained from 
administrative records maintained by State Departments of Education. 

Community Eligibility Provision (CEP) 

The Flealthy, Plunger Free Kids Act of 2010 made the Community Eligibility Provision (CEP) 
available to Local Education Agencies (LEAs) and schools in high poverty areas. Under the CEP, 
families are not required to submit applications for free or reduced-price meals and schools must 
provide free meals to all students. The potential benefits are that in high-poverty schools more 
students will have access to nutritious meals and LEAs may experience reductions in administrative 
burden. The CEP was phased in starting in the 2011/12 school year in three States and four additional 
States participated in each of the two following years. The CE Provision will be available to all States 
in the 2014/15 school year. Formerly known as the Community Eligibility Option (CEO). 

Consolidated Claims 

Claims for reimbursement combined for all of the schools in the LEA. 

Direct Certification 

Direct certification determines children’s eligibility for free meals by matching student enrollment 
records with administrative records from the Supplemental Nutrition Assistance Program (SNAP), 

Temporary Assistance for Needy Families (TANF), or the Food Distribution Program on Indian 
Reservations (FDPIR). In addition, children can be certified as eligible for free meals based on lists 
maintained by LEAs or other agencies, including homeless, runaway and migrant youth, and foster 


Abt Associates 


Glossary | pg. 171 






Community Eligibility Provision Evaluation: Final Report 


children. Children certified for free meals by any of these means—without applications—are 
“identified students.” 

Direct Certification with Medicaid (DCM) 

A demonstration program administered in select States allowing schools to determine students as 
eligible to receive free meals through the National School Lunch and School Breakfast programs 
based on documentation of participation in the Medicaid program, and without application. 

Eligible Non-Participating (EN) LEA 

Local Education Agencies that have an Identified Student Percentage of over 40 percent and do not 
participate in the Community Eligibility Provision. 

Eligible Participating (EP) LEA 

Local Education Agencies that have an Identified Student Percentage of over 40 percent and are 
participating in the Community Eligibility Provision. 

Eligible School 

A school is eligible for the CEP if the LEA is eligible to participate LEA-wide, or if the school has an 
ISP of 40 percent or more, or if the school is grouped with other schools so that the ISP for the group 
is 40 percent or more. 

Enrollment 

The number of students registered at the school or district. 

Food and Nutrition Service (FNS) 

The division of the USDA that administers the School Lunch Program and other Child Nutrition 
Programs. 

Food-Based Menu Planning 

Menu Planning based on food groups, rather than nutrients. These food groups include Fruits, 
Vegetables, Grains, Meat/Meat Alternates and Milk. 

Foodservice Director 

The individual responsible for the school meal operations in an LEA. 

Foodservice Management Company (FSMC) 

A private organization that operates school foodservice under contract with a school district. 

Free and Reduced Price Meal Applications 

Forms completed by households to determine student eligibility for free or reduced-price meals in the 
National School Lunch and School Breakfast Programs. The forms collect information related to 
household income and the number of persons in the household. 

Free Meals 

School meals served to eligible students whose family incomes are below 130 percent of the poverty 
line. 

Full Priced Meals 

School meals served to students whose family income is above 185 percent of the poverty level (or 
whose families do not apply for assistance). 
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Grab-and-Go 

Pre-packaged, or bagged meals, that offer all food components or food items in the quantities required 
for each grade group in an entirely or partially pre-packaged manner. 

Healthy, Hunger Free Kids Act of 2010 (HHFKA) 

The Act authorizing funding and policy reforms for the United States Department of Agriculture, 
including provisions related to the National School Lunch and School Breakfast Programs. Of interest 
to this evaluation, the Act provided the Community Eligibility Provision and also made modifications 
to school menu planning, introducing the New Meal Pattern. 

Identified Student Percentage (ISP) 

To be eligible to participate in the Community Eligibility Provision, an LEA, a group of schools in an 
LEA, or an individual school must have at least 40 percent of its student body identified through 
direct certification and/or inclusion on other agency lists (e.g., migrant, homeless, or foster care 
youth). This group of identified students is referred to as the “identified student percentage.” 

In-classroom Breakfast/Lunch 

Meal service where food is brought to the classrooms from the cafeteria in containers and distributed 
to students or served from carts in the hallways by foodservice staff. 

Indirect Costs 

LEA pro-rated costs distributed across multiple programs, e.g., utilities. 

Individual School Claim 

A claim for reimbursement for a specific school. 

Local Education Agency (LEA) 

An LEA is basically synonymous with a school district. LEAs are often made up of several public 
schools in a district, but they can also be an individual charter or private school. 

Meal Count Day 

The specific day for which meal counts are provided. Ideally, this will be the last school day before 
the visit. 

Meal Counts 

A compilation of the number of meals served by category, including free, reduced-price and paid 
meals. 

Meal Count Types 

The variety of methods employed for counting meals in schools, these may include manual cash 
registers, manual ticket systems, roster check off systems and electronic point of sale (POS) systems. 

Menu Survey 

The Menu Survey collected information about foods offered to students for breakfast and lunch each 
day of the target week. The Menu Survey Booklet included three sections: the Daily Meal Counts 
Form, the Reimbursable Food Form (Breakfast and Lunch Forms), and the Self-Serve/Made-to-Order 
Bar Form. The Recipe Forms, included in a separate booklet, were also part of the Menu Survey 
materials. The data gathered from these surveys was used to analyze the nutrient content and quality 
of meals offered to students. 
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National School Lunch Program (NSLP) 

The National School Lunch Program (NSLP) is a Federally assisted meal program operating in public 
and nonprofit private schools and residential child care institutions. It provides nutritionally balanced, 
low-cost or free lunches to children each school day. 

Near-Eligible (NE) LEA 

Local Education Agency with an Identified Student Percentage between 30 and 40 percent. 

New Meal Pattern 

The Healthy, Hunger Free Kids Act of 2010 established a new food-based meal pattern with grade- 
specific modifications to food quantities, requirements for whole-grains and for specific vegetable 
types to be offered each week. The New Meal Pattern also includes weekly dietary specifications 
including grade-specific calorie ranges, sodium targets, limits on saturated fat, and daily limits on 
trans fat. The New Meal Pattern for the National School Lunch Program was implemented in school 
year 2012/2013, with interim requirements for whole grain rich foods and sodium. Schools were not 
required to adopt the New Meal Pattern for the School Breakfast Program in the 2012/2013 school 
year, with most schools continuing the Traditional Food-Based Menu Planning approach (see below) 
for breakfast. 

Non-CEP Schools 

Schools which are not using the Community Eligibility Provision. These are the comparison schools 
for this evaluation. 

Observation Day 

The specific day of the Target Week when a school is visited. 

Offer-versus-Serve (OVS) 

The option of allowing students to decline some of the food offered in a reimbursable lunch or 
breakfast. The goals of OVS are to reduce food waste and to permit students to choose the foods they 
want to eat. OVS must be offered by high schools but it is optional for elementary or middle schools 
to offer this option. 

On-Site Production Systems 

Meal preparation system occurring in the school building where the meals are served. 

Personally Identifiable Information (PII) 

PII is information that can be used to uniquely identify, contact or locate an individual. Examples 
include last names, addresses, telephone numbers, signatures, and Social Security Numbers. 

Point of Sale (POS) 

The point in the foodservice operation where a determination is made that a meal is eligible for 
reimbursement and is counted as such. This point is generally at the end of the service line. Most but 
not all schools use an electronic device at the POS but some use simple cash registers or possibly 
even cash boxes. 

Provision 2 and 3 

Provisions 2 and 3 are alternatives to the normal requirements for annual determinations of eligibility 
for free and reduced price school meals and daily meal counts by type (free, reduced price and paid 
meals) at the point of service. Provision 2 and 3 reduce application burdens and simplifies meal 
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counting and claiming procedures. Provision 2 allows schools to establish claiming percentages and 
to serve all meals at no charge for a 4 year period. Provision 3 allows schools to simply receive the 
same level of Federal cash and commodity assistance each year, with some adjustments, for a 4 year 
period. 

Record Abstractions 

Collection of information from school Free and Reduced-Price Meal Applications. 

Reduced Price Meal 

School meals served to students whose household income is between 130 percent and 185 percent of 
the poverty level. 

Reference Month/Week/Day 

The month, week or day for which a given piece of information is provided. For example, the LEA 
will provide an enrollment figure for a specific month, week or day. 

Reimbursable Foods 

Food items eligible for inclusion in a reimbursable meal, these exclude foods of minimal nutritional 
value. 

Reimbursable Meal 

A meal that meets the meal pattern requirements. If a school has the OVS option, the meal selected by 
the student must include the minimum components required according to the selected menu planning 
system for schools to be reimbursed by the USD A. Schools receive some reimbursement from the 
USDA for each meal served, whether it is full-price, reduced-price, or free. 

Reimbursement 

The transfer of funds from one agency to another, to offset the costs of operating a program. In the 
NSLP, reimbursement is based on specified dollar amounts for paid, reduced price, and free lunches. 
CEP Schools are reimbursed for all meals based on their established claiming percentage. 

Respondent 

The specific individual from whom information is to be collected for each data collection instrument. 

Rosters 

Lists of individuals, these may include staff members for the Administrative Cost Interviews or 
students for meal count purposes. 

School Breakfast Program (SBP) 

The School Breakfast Program (SBP) provides cash assistance to States to operate nonprofit breakfast 
programs in schools and residential childcare institutions. The program is administered at the Federal 
level by FNS. State education agencies administer the SBP at the State level, and local school food 
authorities operate it in schools. 

Self-Serve 

Foods which students may serve or portion for themselves, such as salad bars or condiment bars. 

Some schools also offer other theme bars which may be partially self-serve (e.g., toppings for potato 
bars or taco bars). 
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Shared Cafeteria 

Two schools that share a location for their students to eat meals, as well as a kitchen for preparing 
meals. Students from the schools may be served either during separate meal periods, or at the same 
time. 

School Food Authority (SFA) 

The governing body with the legal responsibility for administering the National School Lunch and/or 
School Breakfast Program in one or more schools. In this evaluation, the term Local Education 
Agency is used in place of SFA. 

Target Week 

The target week is the pre-designated five-day period in which onsite data collection took place in 
Component 2 and 3 LEAs. The target week is also the week for which the menu survey data were 
collected in Component 3 schools. 

Traditional Line Service 

Meal service model where students wait at point of service for school food to be served. 

U.S. Department of Agriculture (USDA) 

Supervises the National School Lunch and School Breakfast Programs, setting meal standards and 
providing schools with reimbursements for meals served. 

Vended Meals 

Meals purchased by schools from a third party. The vendor is generally a private company, but in 
some circumstances it may be another school district. 

Verification Summary Report (VSR) 

The Verification Summary Report provides data on State, number of schools, total enrollment, 
percentage of students approved for FRP meals, ISP, and number of students in non-base-year 
Provision 2/3 schools. Also known as the FNS School Food Authority Verification Summary Report 
(FNS-742). 
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Appendix 2A: Constructing the Sampling Frame 


This appendix presents supporting information on the sampling frame for the CEP evaluation. First, 
data sources and merging procedures are discussed. Next, sample exclusions and eligibility for the 
components of data collection are explained. 

2A.1 Data Sources and Merging Procedures 

The evaluation team constructed the sampling frame by merging data from three sources: 

• Common Core of Data (CCD) 

• Verification Summary Report (VSR) data 

• State-provided data files 

These sources are described next. 

2A.1.1 Common Core of Data 

The Common Core of Data (CCD), compiled by the National Center for Education Statistics, is a 
longitudinal database of key characteristics of LEAs and schools. Most of the data are obtained from 
administrative records maintained by State Departments of Education. In addition to regular public 
school and charter school LEAs, the CCD includes regional education service agencies, State and 
Federally operated agencies, and other types of public education agencies. However, data for these 
latter types of LEAs can be incomplete. Data for the preceding school year are typically released in 
the early-to-mid spring of each year. The most recent CCD data available were used to construct the 
sampling frame for primary data collection and the Matched Administrative Data Sample. Because 
sampling for primary data collection was completed in the summer of 2012, the most recent data were 
drawn from the 2009-10 school year. Sampling for the Matched Administrative Data Sample was 
conducted in the spring of 2013 and utilized data from the 2010-11 school year. CCD data for LEAs 
and schools, respectively, are downloadable from http://nces.ed.gov/ccd/pubagency.asp and 
http://nces.ed.gov/ccd/pubschuniv.asp . 

2A.1.2 Verification Summary Reports (VSR) 

The VSR database is constructed from FNS-742 reports submitted by LEAs to State agencies, which 
are in turn compiled by FNS. The VSR data contain a wealth of information, including data on 
student enrollment, number of schools, and numbers of students eligible for free or reduced-price 
meals under different eligibility categories. These data are based on the annual official counts of 
students eligible for free or reduced-price meals as of October 30. FNS provided the 2011-12 VSR 
data for the CEP States to the evaluation team, who then relied on these data for LEA enrollment, the 
percentage of students eligible for free or reduced-price meals (FRP), and the Identified Student 
Percentage (ISP). 

2A.1.3 State Data 

The evaluation team requested data on a number of LEA and school characteristics from State CN 
agencies. Most importantly, the evaluation team asked States to provide ISP and CEP participation 
data for both LEAs and schools, in order to identify CEP eligibility and participation status for LEAs 
and schools. FNS policy specified that ISP data as of April 1, 2012, should be used for determining 
eligibility for the CEP for the 2012-13 school year. 
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In addition to the State data described above, the Matched Administrative Data Sample included 
administrative data on participation, enrollment, attendance and revenue (as described in section 
2.4.1) in the propensity score model. These data were included to insure that the sample was balanced 
on baseline measures of NSLP meal counts, but did not affect the construction of the sampling frame. 

2A.1.4 Data Merging 

As States delivered data, the evaluation team cleaned and integrated these fdes with its existing 
datasets. The composition and comprehensiveness of these files varied by State, but all States 
provided the most essential elements for the Impact Study—school and LEA participation in the CEP 
for the 2012-13 school year. Rosters of participating schools were used to determine the percentage 
of schools participating in each LEA as well as the components of data collection for which each 
LEA would be eligible. 

These data were merged with the 2011-12 VSR and the 2009-10 CCD, which provided 
characteristics that were used to determine sample inclusion, as discussed in more detail below. 

Data from States that implemented the CEP in 2011-12 were merged to a fourth data source as well: 
the 2010-11 VSR. For these States, the October 2010 VSR data are used to construct measures that 
enter the propensity score model, as they are the relevant data from the year immediately preceding 
implementation of the CEP. If an LEA in a Year One State was missing ISP or FRP data in the 2010- 
11 VSR, the data were pulled instead from the 2011-12 VSR. 

States also provided contact information for LEA foodservice directors (FSDs). These data were used 
to identify potential duplication among FSDs within a State. The sampling issues arising from any 
such duplication of FSDs are further discussed below (see the heading “Foodservice Director 
Duplication”). 

2A.2 Sample Exclusions 

2A.2.1 LEAs Serving Special Populations 

The Impact Study excludes LEAs that exclusively serve special populations, such as students with 
severe disabilities, because (1) such LEAs are more likely to include residential schools than other 
public school LEAs, and residential programs are not eligible to participate in the CEP, and (2), the 
foodservice operations in such LEAs’ non-residential programs may operate quite differently than 
those in other public school LEAs, introducing variables that would likely confound comparisons 
with more typical LEAs. State- and Federally operated LEAs, including juvenile detention centers, 
and regional education agencies serving special populations, were similarly excluded from the Impact 
Study. 

2A.2.2 Private Schools 

The Impact Study sample also excluded private schools, because there was no known data source that 
could provide the elements needed to find appropriate comparison schools. (The CCD includes only 
public schools.) Non-public LEAs constitute approximately 15 percent of participating LEAs. Non¬ 
public LEAs are smaller, on average, than public LEAs, and therefore, because the proportion of non- 
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public LEA students experiencing the CEP is likely to be considerably less than 15 percent, exclusion 
of private schools is a minor limitation of the study. 1 

2A.2.3 LEAs Missing from the 2009-10 CCD or 2011-12 VSR 

The Impact Study sample excludes LEAs that do not appear or appear as non-operational in the 
2009-10 CCD and LEAs that do not appear in the 2011-12 VSR data. These exclusions have three 
purposes. First, requiring the LEA to be operating in 2009-10 eliminates LEAs that have opened 
recently and therefore do not have sufficient pre-intervention data for inclusion in the CITS design. 
Second, using the 2011-12 VSR data ensures that the LEA was in operation in the most recent school 
year. Third, this ensures that sufficient data are available for identifying comparison group LEAs, 
through the propensity score matching process. 

2A.2.4 Direct Certification with Medicaid Study Treatment LEAs 

The selection of the impact sample in Illinois was accelerated due to the need to coordinate with the 
Direct Certification with Medicaid (DCM) evaluation. FNS and its contractor for the DCM evaluation 
were planning to randomly assign a sample of Illinois LEAs to participate in DCM. This woidd 
confound the impact of the CEP on administrative costs and errors, because LEAs using Medicaid for 
direct certification (in addition to SNAP and TANF) would process fewer FRP meals applications. 
FNS agreed to allow the CEP to select its sample in Illinois first and to exclude the sampled LEAs 
from the DCM evaluation. It was, however, only feasible to select the Year 1 sample for Illinois 
(SY 2011-12 CEP LEAs and their comparison LEAs) prior to the random assignment of LEAs to the 
DCM treatment group. The evaluation team provided the Year 1 Illinois sample to FNS on May 18, 
2012. FNS provided the sample of all Illinois LEAs assigned to the DCM treatment group on August 
8, 2012. LEAs were excluded from the CEP evaluation if they were selected for the DCM treatment 
group. A total of 350 such LEAs were excluded. Due to the random assignment used by the DCM 
evaluation, this exclusion was not expected to reduce the representativeness of the Illinois sample. 

Two other States were included in both the DCM evaluation and the CEP evaluation, but sampling 
was unaffected in both States. Kentucky implemented DCM statewide, so there was no effect on the 
sampling. New York implemented DCM in randomly selected schools within the New York City 
Board of Education (NYCBOE) system, which was already excluded from the CEP evaluation 
because it was not feasible to match the NYCBOE with another non-participating LEA in New York. 

2A.2.5 APEC-II Study Sample 

Sampling for Components 2 and 3 of the CEP Evaluation was coordinated with the Access, 
Participation, Eligibility and Certification (APEC)-II study because the field periods for the studies 
overlap, and both studies require reviews of certification records. The overlap of data collection was 
likely to overburden LEAs and potentially lead to biased results if LEAs “cleaned up” their records 
after one study visited. The APEC-II study needed to select two samples: the main (nationally 
representative) sample and a supplementary sample of LEAs operating under the CEP. The main 
APEC-II sample excluded LEAs fully participating in the CEP, but potential for overlap between the 
two samples existed both among LEAs partially participating in the CEP and among non-participating 
LEAs that were selected for the comparison group for the CEP Evaluation. 


1 Due to limitations of enrollment data for private schools, the ratio of private to public school students 
among all participating LEAs cannot be computed with confidence. 
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To facilitate coordination of sampling for the two studies, FNS provided the primary and back-up 
selections for the main APEC-11 sample in the CEP States. (Coordination of sampling in the District 
of Columbia was not needed, because the District was already excluded from the CEP Impact Study.) 
The evaluation team compared this list with the sampling frame for the CEP Evaluation, including the 
participating LEAs and their two best matches as comparison LEAs. The two studies coordinated 
sampling so that, where there was overlap, the APEC-11 study could use the primary selections for its 
main sample (as long as they were not participating in the CEP in SY 2012-13) while the CEP 
Evaluation could include LEAs that were needed for the Component 2 and 3 samples (including 
partially-participating treatment LEAs and comparison LEAs). During the recruiting phase, 7 LEAs 
were released from the CEP sample and made available to the APEC-11 study, including 3 LEAs that 
had been selected as backups, 2 that were unresponsive to recruiting efforts for the CEP Evaluation, 
and 1 that shared a foodservice director with multiple LEAs in the impact sample. The 2 LEAs that 
were dropped from the primary sample were replaced with suitable backups. 

2A.2.6 LEAs Offering Alternative Universal Free Meals Programs, Including Provisions 2 
and 3 

LEAs using Provisions 2 and 3 offer free meals to all students and count only the total meals, then 
claim reimbursement using percentages or dollar amounts established in base years when students are 
certified for FRP meals. Therefore, the effects of Provisions 2 and 3 on student participation in NSLP 
and SBP are likely to be similar to those of the CEP. As a result, the study would likely be unable to 
differentiate between participation under Provision 2 or 3 and under the CEP. Further, those LEAs 
subject to Provision 2 or 3 do not represent the desired counterfactual condition of conventional 
reimbursement procedures. To overcome this challenge, the evaluation excluded all LEAs with any 
Provision 2 or 3 schools from the Impact Study sample. The Impact Study also excluded LEAs with 
any Provision 2 or 3 schools in the year prior to implementation of the CEP, as well as LEAs that 
offered universal free meals under alternative programs sponsored by States. Provision 2 and 3 LEAs 
were excluded from both the Matched Administrative Data Sample and the Primary Data Collection 
Sample. 

Flowever, LEAs that operate an alternative universal free meals program were excluded from the 
Primary Data Collection Sample, but remained in the Matched Administrative Data Sample. While 
Provision 2 or 3 status is observed in available FNS data, State-sponsored universal free meals 
programs do not appear in Federal data sources. West Virginia provided State data that identified 
LEAs offering alternative universal free meals programs. In other States, LEAs that offered universal 
free meals programs coidd only be screened out after the initial sampling process. Among LEAs 
selected to participate in on-site data collection, the recruiting process screened out those offering 
universal free meals through an alternative program. Among LEAs selected to participate only in the 
PEAR survey, LEAs that participate in an alternative universal free meals program were screened out 
as part of the survey. 

2A.2.7 One-Year Only Participants 

The final sample exclusion is of LEAs that took up the CEP in 2011-12 but did not to continue 
participation in the 2012-13. There were three such LEAs, all from Michigan. 
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2A.2.8 Foodservice Director Duplication 

Occasionally, different LEAs share the same foodservice director (FSD), and although most such 
instances represent pairs of LEAs, some represent other situations. Specifically, when more than two 
LEAs share an FSD, State-level respondents have indicated that: 

1. The LEAs use the same foodservice management company, and the regional manager is the FSD 
for the group of LEAs. 

2. The LEAs are charter schools run by the same parent organization (e.g., Eleritage Academies), 
and the management of foodservice is centralized at the parent organization. While the LEAs 
have separate agreements for NSLP/SBP with the State and separate meal claims, they function 
largely as if they were part of the same SFA. 

There are two problems that arise in these situations with shared FSDs. First, responding to multiple 
surveys and interviews would represent an excessive burden for one FSD. Second, it appears that 
some or all of the relevant processes (certification, menu planning, meal claiming) are not 
independent across LEAs with a shared FSD. Therefore, the study gains power to detect impacts by 
including as many LEAs with unique FSDs as possible. 

For these reasons, as noted in prior communication with and approval from FNS, the evaluation team 
modified the sampling frame for primary data collection in both the Implementation Study and the 
Impact Study. (The Matched Administrative Data Sample included LEAs irrespective of shared 
FSD.) The team identified groups of participating LEAs with the same FSD in each State and selected 
two LEAs to be included in the sampling frame, randomizing within a set of preferences that 
prioritized participation in the CEP, as well as size and grade coverage.” Thus, any one FSD can be 
selected to participate in the evaluation at most for two LEAs within a State/ 

2A.2.9 LEAs in the District of Columbia 

With FNS approval, the evaluation team excluded the District of Columbia (DC) from the Impact 
Study. The evaluation team made this recommendation after a review of LEA data for DC, but before 
obtaining lists of participating LEAs. This decision was made for two main reasons. First, DC has a 
unique structure, with only one regular school district, thus limiting the potential sample for the 
Impact Study to charter schools. This structure made Component 3 data collection impossible and 
raised concerns regarding the number of potential LEAs available for inclusion in Components 1 and 
2. Second, DC did not require schools to choose to take up the CEP until the end of September, 2012. 
This would have delayed the sampling efforts significantly; and since DC could only contribute a 
marginal number of matched pairs, the evaluation proceeded without DC. 2 * 4 


2 Those LEAs with shared FSDs were identified by using an algorithm that matched on name spellings 
within a defined tolerance limit. The algorithm matches strings based on their Levenshtein edit distance 
(Reif, 2012). 

During data collection, one FSD was identified as the respondent for LEAs in multiple States. Among 
these LEAs, the evaluation team selected the ones sampled for the most data collection. 

4 LEAs in the District of Columbia remained eligible for the Implementation Study. 


Abt Associates 


Appendix 2A | pg. 2A-5 





Community Eligibility Provision Evaluation: Final Report 


2A.2.10 Component Eligibility 

LEA Eligibility for Matched Administrative Data Sample 

The Matched Administrative Data Sample was used to measure the impact of the CEP on key 
participation and revenue outcomes using State administrative data. All LEAs eligible for the Impact 
Study were eligible for the Matched Administrative Data Sample. LEAs with at least one 
participating school were considered treatment LEAs, and LEAs with no participating schools were 
considered potential comparison LEAs. 

LEA Eligibility for Component 1 

Component 1 consisted of PEAR survey respondents. All LEAs eligible for the Impact Study were 
eligible for Component 1. LEAs with at least one participating school were considered treatment 
LEAs, and LEAs with no participating schools were considered potential comparison LEAs. 

LEA Eligibility for Component 2 

Component 2 included on-site data collection: cost interviews and certification record reviews. A 
non-participating LEA was eligible for Component 2 if it had at least one eligible school, as defined 
in the section on school eligibility below, and the total enrollment in all eligible schools was at least 
300. The enrollment requirement was to ensure that there are sufficient records for certification record 
reviews. 

A participating LEA was eligible for Component 2 if it had sufficient enrollment, a school eligible for 
on-site data collection, and 70 percent or more of eligible schools participating in the CEP. The 
significant participation requirement was selected to best use the limited resources of the study. This 
component compared the costs and accuracy of the certification process when applications are not 
required to those under conventional program operations. Therefore, the power to detect impacts was 
stronger because the evaluation included only LEAs that were largely characterized by the CEP 
experience. LEAs that were almost-fully participating (70 percent or more) were included to ensure a 
sufficient sample. 

LEA Eligibility for Component 3 

Component 3 consisted of an on-site menu survey and cashier observations. LEA eligibility criteria 
for this component were identical for both participating and non-participating LEAs: having three 
schools eligible for on-site data collection that each represents a distinct grade range and together 
span kindergarten to 12 th grade. 

Menus are likely to differ substantially by grade level in keeping with current NSLP standards. 
Therefore, for Component 3 sampling, the objective was to include only LEAs with schools covering 
three grade ranges: elementary (grades K-5), middle (grades 6-8), and high (9-12). (Schools with 
grades covering multiple levels (e.g., grades K-8) were treated as a member of either level.) This 
requirement considerably restricted the sampling frame for the Component 3 sample, because 
relatively few LEAs had more than one high or middle-high school, and many LEAs had none. For 
example, more than half of participating LEAs in Illinois and Ohio had no high or middle-high 
schools, as did somewhat more than a third of LEAs in Michigan and New York. 


pg. 2A-6 | Appendix 2A 


Abt Associates 




Community Eligibility Provision Evaluation: Final Report 


School Eligibility for Components 2 and 3 

For Components 2 and 3, schools were eligible for on-site data collection based on 2009-10 CCD 
data if the school (1) was defined as either a “regular school” or a “vocational school”, (2) was open 
and operational, and (3) had enrolled students. 
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Appendix 2B: Constructing the Matched Administrative Data 
Sample 


The evaluation team used propensity score matching to identify matched pairs of participating, 
“treatment,” LEAs and non-participating, “comparison,” LEAs. The evaluation team estimated a 
statistical model of participation and generated a “propensity score.” This score is the LEA’s 
predicted probability of taking up the CEP, which can be used to identify similarly-likely non-takers 
whose outcomes will serve as a reasonable counterfactual. For the Administrative Data Sample, that 
counterfactual includes, with very few exclusion, those LEAs that did not take up the CEP but that 
seem like ones that did. This is considered a “business as usual” comparison: those in the 
comparison group are used to represent the world as it is without the CEP but with any other services 
or programs that might exist in schools and LEAs to provide school meal-related assistance. This 
appendix describes the statistical model of participation used to generate the propensity score and the 
procedure used to match the LEAs. 

2B.1 Propensity Score Modeling 

2B.1.1 Variables Included in Model 

The dependent variable in the model was a binary variable that took on a value of one if the LEA had 
at least one participating school in 2012-13 and zero otherwise. All States were estimated together 
using logistic regression. Because the direct certification process and LEA structures differed 
considerably across CEP States, the evaluation team anticipated that the relationship between 
observed characteristics and participation might also vary across these States, and so the models 
included State-specific characteristics and indicators where possible. 5 Note that in contrast, the 
matching procedure for primary data collection used separate regression models for each State. 

The propensity score modeling for the Matched Administrative Data Sample was performed in the 
spring of 2013 and built on the model developed for sampling for the Primary Data Collection 
Sample. See Appendix A.2.c for a discussion of the development of that model. The list of variables 
for inclusion was adjusted for two reasons. 

First, the propensity score model for the Matched Administrative Data Sample was developed to 
investigate factors related to take-up of CEP. Flence, the team transformed variables to minimize 
multi-collinearity, which allows the reader to interpret the coefficients more readily. Hence, the 
Matched Administrative Data Sample is based on a propensity score model that includes the 
difference between FRP and ISP, rather than FRP. Since FRP is highly correlated with ISP, including 
both would make the standard errors larger. The coefficient of the difference between FRP and ISP 


5 For a description of how direct certification differs across States, see the 2012 Direct Certification Report 
available at http://www.fns.usda.gov/Ora/menu/Published/CNP/FILES/DirectCert2Q12.pdf . As direct 
certification determines an LEA’s Identified Student Percentage (ISP) and ISP determines the 
reimbursement rate an LEA would obtain if it took up the CEP, differences in direct certification processes 
are likely to affect participation. In particular, in States that devolve responsibility for direct certification to 
the local level, LEAs may have a greater ability to manipulate their reimbursement ISP which changes the 
relationship between participation and the pre-intervention ISP included in the propensity score model. 
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can be interpreted to measure the effect on take-up of CEP of increasing FRP while holding ISP 
constant. 

Second, the propensity score model for the Matched Administrative Data Sample included pre¬ 
intervention measures of meal counts. Because an LEA’s meal counts are likely to be highly 
correlated with both take-up of the CEP and with post-intervention meal counts, including these 
variables in the model strengthens the design. However, these measures were not available in the 
summer of 2012 and therefore could not be included in the propensity score model for primary data 
collection. 

The following variables were included in the propensity score model for the Matched Administrative 
Data Sample. 

• ISP categories (between 20-30, 30-40, 40-50, 50-60, 60-70, >70) 

- Reference category <20 percent 

• Distribution of percentage of meals taken as free and reduced (FRP) 

- Percentage of meals taken as free (less than 50 percent) and reduced (less than 10 percent) 

- Percentage of meals taken as free (more than 50 percent) and reduced (less than 10 percent) 

- Percentage of meals taken as free (less than 50 percent) and reduced (more than 10 percent) 

- Reference category - Percentage of meals taken as free (more than 50 percent) and reduced 
(more than 10 percent) 

• Difference between FRP and ISP 

• LEA characteristics 

- Percentage of students who are English language learners (ELL) 

- Percentage of students who are in grades K-5 

- Percentage of students who are in grades 6-8 

- Whether the LEA is in an urban area 

- Log number of schools 

- Mean school enrollment in an LEA <350 

- Percentage of students who are Black 

- Percentage of students who are Hispanic 

- Percentage of schools which are Title 1 schools 

• Charter indicator for Ml, NY and OH 

• State Dummies (Reference category is KY) 

The rationales for these variables and details of their definitions are discussed below. The ISP, FRP 
and the difference between the two came from pre-implementation VSR data: the 2010-11 VSR data 
for Illinois, Kentucky and Michigan and the 2011-12 VSR data for New York, Ohio and West 
Virginia. All other data elements were from the 2010-11 CCD. Additional discussion of other 
important independent variables is below. 
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2B.1.2 Definition of ISP Used in Modeling 

The LEA’s ISP was included as an independent variable because it is the key variable determining 
CEP eligibility. The evaluation team defined ISP as follows for the propensity score model: 6 

Number Free Eligible Not Subject to Verfication + Number Categorically Eligible 

Enrollment 

This variable was used only as a proxy for true ISP for propensity score modeling purposes. While 
technically only the students who are not subject to verification should be included in the ISP, the 
definition above reproduced more closely the reimbursement ISPs reported by participating LEAs. 
When an LEA elects to participate in the CEP, it can revisit certification data to identify additional 
categorically eligible students approved by application who can be directly certified and thereby 
increase its reported ISP. 7 

2B.1.3 Additional Variables 

The distribution of FRP percentage is included in the model as a broader measure of LEA poverty, to 
capture the foodservice context in the LEA and to disentangle the relationship with ISP since they are 
highly correlated. The omitted reference category is those LEAs with greater than 50 percent 
qualifying for free lunch and greater than 10 percent qualifying for reduced lunch. The difference 
between FRP percentage and ISP allows for interaction between the two measures in the model. 

To avoid allowing a few very large LEAs to determine the regression coefficients for measures of 
LEA size (number of schools and enrollment), the logarithmic transformation of the number of 
schools is used. Enrollment is highly correlated with number of schools, so the model includes a 
dummy variable for school size less than 350. The omitted reference category is the school size 
greater than 350. 

In the initial exploratory interviews, State Child Nutrition (CN) directors indicated that some LEAs 
were reluctant to implement the CEP at the secondary school level, and that therefore the CEP was 
more likely to be implemented in elementary schools. The evaluation team therefore includes the 
percent of students in kindergarten through grade 5 and the percent of students in grades 6 through 8 
in the model. The omitted reference category is the percentage of students in grades 9 through 12. 
Also, LEA characteristics that were seen to be highly correlated to participation were added to the 
model; these included percent of students who are Black, percent of students who are Latino, percent 
of students who are ELL, and percent of schools that are Title I. Section 4.1 also shows that 
participating LEAs tend to be urban LEAs, and so a dummy for an LEA in an urban area was also 
included in the model. 


6 Note that the evaluation team excludes all LEAs that operate Provision 2 or 3 schools and therefore does 
not need to exclude these students from the denominator. ISP calculations for the Implementation Study 
exclude students who are categorically eligible by application from the numerator and the Provision 2 or 3 
students from the denominator. 

7 The extent to which these numbers were manipulable varied across States, according to State officials who 
manage the collection and maintenance of nutrition data. 
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2B.1..4 Charter Schools 

The evaluation team allowed the model of charter school participation to vary substantially across 
States, reflecting the variation in the way charter schools are structured. Kentucky and West Virginia 
have no charter schools. In Illinois, charter schools are members in a regular public school LEA and 
thus part of its foodservice operations. In Michigan, New York, and Ohio, most charter schools 
operate as distinct, often single-school LEAs, and appear separately in the VSR. 

For the three States where charter schools function as separate LEAs for foodservice purposes, 
preliminary analyses demonstrated that the participation rate and the factors associated with the CEP 
participation of charter school LEAs differed substantially from regular LEAs within the same State. 
In Michigan, the CEP participation rate among charter schools was 36 percent, compared to 12 
percent among traditional LEAs. In Ohio, the participation rate among charter schools was 19 
percent, compared to 2 percent for traditional LEAs. In New York, the participation rate among 
charter schools was 13 percent compared to 1 percent among traditional LEAs. 8 

Because the prevalence of charters differs greatly across these States, the propensity score model 
included separate dummy variables for charter schools in Michigan, Ohio and New York. 

2B.2 Propensity Score Matching Procedure 

As discussed earlier in the Impact Study Design, the evaluation team matched the pool of eligible 
LEAs using one-to-five matching with replacement. This matching procedure allows up to five non- 
participating LEAs to provide information on the counterfactual scenario for each participating LEA. 
Further, each non-participating LEA is allowed to be matched an unlimited number of one 
participating LEA. The matching procedure constructs weights for each comparison LEA that reflect 
the number of treatment LEAs for which it was selected to serve as comparison and how many 
comparisons were identified for each of those treatments. To prevent the use of non-comparable 
LEAs as matches, the evaluation team required the treatment and comparison LEAs to have 
propensity scores within 1 percentage point of one another. 

Results of the propensity score modeling and matching are discussed below. 

2B.3 Propensity Score Modeling and Matching Results 

2B.3.1 Regression Results 

The results from the logistic regressions for CEP participation in these States are summarized in 
Exhibit 4.10 in the main text. See Section 4.3.2 for a detailed discussion on the interpretation of the 
results of the logistic regression. It is important to note that this model differs from the models 
developed for on-site data collection as this model seeks to interpret or explain how the dependent 
variable is related to the independent variables. (Conversely, the models for primary data collection 
were constructed with the purely mechanical objective of capturing as much of the variation in 
participation as possible by maximizing the Pseudo-R squared and were done separately for each 
State to facilitate data collection). 


These participation rates are calculated for LEAs that are eligible for inclusion in the Impact Study. 
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Because the goal was interpretability, it was not possible to include some variables (at least in their 
continuous form) due to high multi-collinearity with other independent variables. For example, ISP 
and percent FRP are two measures of poverty that are highly correlated with urbanicity and LEA size. 
Multi-collinearity poses no threat to maximizing the Pseudo-R squared, but it does render 
interpretation of coefficients problematic. As a result, the analysis produced indicator variables 
representing the distributions of several continuous variables, in order to improve the interpretability 
of the results. See Section 4.3 for a complete discussion of implication of the logistic regression. 

2B.3.2 Matching Results 

In total, 306 LEAs participating in the CEP were eligible for the propensity score matching. 9 Out of 
these, 286 participating LEAs were matched to at least one non-participating LEA. 10 Six of the 
unmatched participating LEA had a propensity score lower than the maximum propensity score of the 
non-participating LEAs, but the matching procedure could not find a suitable match. For the 
remaining 14 participating LEAs, their propensity score were higher than the maximum propensity 
score of the non-participating LEAs. These 20 LEAs were dropped from the analysis. Ninety-one 
percent of participating LEAs had at least three or more matches to non-participating LEAs. In total, 
286 participating LEAs were matched to 525 non-participating LEAs across six States. Exhibit 2.6 
presents the comparison of the treatment and comparison group characteristics and shows that the two 
groups have no differences except in baseline SBP participation. 


One LEA in Michigan was excluded in the logistic regression as it was missing information on the number 
of schools. 

10 Ten participating LEAs in West Virginia, three participating LEAs in Kentucky and Michigan and four 
participating LEAs in Illinois did not find a match. 
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Appendix 2C: Constructing the Primary Data Collection Sample 


As in the process used to construct Matched Administrative Data Sample, the evaluation team used 
propensity score matching to identify matched pairs of participating, “treatment” LEAs and non¬ 
participating, “comparison” LEAs for the primary data collection sample. The evaluation team 
estimated a statistical model of participation and generated a “propensity score;” this score is the 
LEA’s predicted probability of taking up the CEP. In the case of the primary data collection sample, 
the matched comparison group was constructed to reflect what one might think of as a “no services” 
comparison group. That is, the group excluded any LEAs that offered other free meal programs, such 
that the resulting comparison was between those LEAs that implemented the CEP and those as- 
similar-as-possible LEAs where no such program was in effect. This appendix describes the 
statistical models of participation used to generate the propensity scores. 

2C.1 Propensity Score Matching 

2C.1.1 Variables Included in Models 

The dependent variable in all models was a binary variable that took on a value of one if the LEA had 
at least one participating school in 2012-13 and zero otherwise. All models were estimated using 
logistic regression. Because the direct certification process and LEA structures differed considerably 
across CEP States, the evaluation team expected that the relationship between observed 
characteristics and participation would also vary across these States. As the structure of primary data 
collection required pairs of treatment and comparison LEAs to be matched within States, propensity 
score models were estimated separately across States. 

Due to the timing of data collection, the primary data collection sample was constructed in the 
summer of 2012. This section describes the development of the original propensity score model 
developed to describe take-up of CEP. The propensity score model used to construct the Matched 
Administrative Data Sample is also based on this model. 

The evaluation team selected independent variables that: 

• Were consistently measured across States; 

• Included conceptually important predictors of participation; 

• Captured features found to be significantly related to participation across varied contexts; and, 

• In combination, explained as much of the variation in participation as possible. 

The evaluation team tested several specifications with the goal of identifying the greatest number of 
near matches between the CEP treatment cases and their comparisons. That is, the evaluation team 
aimed for a resulting sample that would be balanced in its pre-intervention characteristics and offer a 
large enough number of comparison matches to provide needed flexibility in the recruitment process. 

It is generally accepted practice in propensity score matching to use a relatively inclusive model, 
where the aim is to have the greatest predictive power, regardless whether any given variable is 
statistically significant. 11 Selected demographic characteristics and contextual variables available in 


11 Stuart, Elizabeth A. (2010). Matching Methods for Causal Inference: A review and a look forward. 
Statistical Science 25(1): 1—21. 
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the CCD were considered for inclusion in the model; specifically, variables were included because 
they had the most explanatory power, when taken together. The following independent variables were 
included in the State-specific propensity score models: 

• Identified Student Percentage (ISP) 

• ISP spline (defined below) 

• Percentage of students eligible for free or reduced price lunch (FRP) 

• Ratio of FRP to ISP (FRP/ISP) 

• Percentage of students who are English language learners (ELL) 

• Percentage of students who are in grades K-5 

• Percentage of students who are in grades 6-8 

• Whether the LEA is in an urban area (city) 

• Log number of schools 

• Log enrollment. 

• Charter indicator 

• Charter interaction terms (model of charter participation varies by State) 

The rationales for these variables and details of their definitions are discussed below. 

The ISP, FRP and the ratio between the two come from pre-implementation VSR data: the 2010-11 
VSR data for Illinois, Kentucky and Michigan and the 2011-12 VSR data for New York, Ohio and 
West Virginia. All other data elements are from the 2009-10 CCD. Additional discussion of other 
important independent variables is below. 

2C.1.2 ISP Spline 

The definition of ISP is based on VSR data and includes categorically eligible students in the 
numerator for the reasons discussed in Appendix B. 

The relationship between ISP and likelihood of participation could be non-linear. For example, in 
exploratory analyses, the evaluation team found that in some States virtually all LEAs with ISPs 
above a certain threshold value were CE participants. Above that level, further increases in ISP had 
no effect on the propensity to participate. 

Spline specifications are appropriate for modeling such relationships. Where a standard regression 
specification of a variable assumes a linear relationship between independent and dependent 
variables, a spline specification may be used to model piecewise linear relationships. The evaluation 
team included spline functions for the ISP in modeling participation for Michigan and Ohio, where 
inclusion of such terms was feasible and improved model fit; the spline function was specified with 
“knots” (points where the slope changes) at 20, 40, and 60 percent. 12 


Thoemmes, Felix J. (2009). The use of propensity scores with clustered data: A simulation study (Doctoral 
dissertation). Available from ProQuest Dissertations and Theses database (UMI No. 3380671). 

12 A linear spline term for a variable X with a knot at K is defined as X k = max{X-K,0}. Inclusion of spline 
terms was feasible in these States because they had a large number of participating LEAs that were 
distributed across the range of ISP categories. Other States had too few participating LEAs across the 
distribution to include these terms. 
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2C.1.3 Additional Variables 

The FRP percentage is included in the model as a broader measure of LEA poverty and to capture the 
food service context in the LEA. The ratio between FRP percentage and ISP allows for interaction 
between the two measures in the model. 

To avoid allowing a few very large LEAs to determine the regression coefficients for measures of 
LEA size (number of schools and enrollment), the logarithmic transformation of the number of 
schools and enrollment is used. 

In the initial exploratory interviews, State Child Nutrition (CN) directors indicated that some LEAs 
were reluctant to implement the CEP at the secondary school level, and that therefore the CEP was 
more likely to be implemented in elementary schools. The evaluation team therefore included the 
percent of students in kindergarten through grade 5 and the percent of students in grades 6 through 8 
in the model. The omitted reference category is the percentage of students in grades 9 through 12. 

2C.1.4 Charter Schools 

As noted previously in Section 2B.1.4, charter schools were structured differently among States. To 
accommodate this variation, the evaluation team developed a State-specific model of charter 
participation for Michigan, Ohio and New York. Variables were selected to describe charter 
participation through the same process as variables were selected for the model as a whole: variables 
were included if they improved the overall model fit. These variables were included as charter 
interaction terms in the propensity score model. 

In Michigan, the high participation rate among charters and the high prevalence of charters within the 
population of LEAs allowed for a rich model of charter participation. All variables that were included 
in the main model were interacted with a charter indicator. In addition, the variables were added to 
specifically capture characteristics on which charters differed from the larger LEA population: 
variables describing the ethnic composition of the LEA, Title 1 and Magnet status. 

In Ohio, participation patterns and prevalence of charter LEAs allowed for a rich model of charter 
participation. Preliminary analyses identified a different set of terms for inclusion in the Ohio model 
of charter participation than were included in the Michigan model. In Ohio, the following variables 
were interacted with a charter indicator and included in the propensity score model: ISP, FRP percent, 
the ratio of FRP percent to ISP, percent of total enrollment in grades K to 5, percent of total 
enrollment in grades 6 to 8, an indicator for whether the LEA was in a city, log enrollment and 
percent Hispanic. 

In New York, the evaluation team included a charter indicator variable in the propensity score model. 
There were too few participating charter school LEAs to support a richer specification. In New York, 
only 13 participating LEAs were eligible for the Impact Study, four of which were charter LEAs. 

2C.1.5 Special Handling of West Virginia 

The evaluation team matched on a single variable (ISP) in West Virginia because sample limitations 
precluded fitting a propensity score model, and the demographic characteristics of participating and 
non-participating LEAs were balanced even without matching on propensity score. These groups 
were initially imbalanced on only two variables: ISP and free and reduced price lunch percentage 
(FRP). 
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In West Virginia, regular, public LEAs correspond to counties, of which there are only 55. Of these, 

18 were excluded from the sampling frame because they offered an alternative universal free meals 
program, leaving only 19 participating and 18 non-participating LEAs. The characteristics of 
participating and non-participating LEAs were balanced on all characteristics included in the 
propensity score model, other than measures of poverty (ISP and FRP percentage). To achieve 
balance on these metrics, the three participating LEAs with the highest ISP values were dropped. The 
procedure described below was then performed by matching on ISP rather than propensity score. 

2C.1.6 Propensity Score Matching Procedure 

The propensity score matching approach finds a similar comparison LEA for each treatment LEA. 
However, the application of propensity score matching to this particular study is complicated by the 
varying eligibility requirements for the various Components of the Impact Study: the evaluation team 
had to find the best match for each treatment LEA among the comparison LEAs eligible for the 
component. Ideally, the sampling frame would be split into three sections, one corresponding to each 
component, and the entire modeling and matching procedure would be applied to each section 
separately. However, the sample sizes were too small to estimate a propensity score model if 
stratifying by both State and component eligibility. Therefore, the evaluation team estimated the 
model for each State without taking component eligibility into account, but stratified by component 
when matching. 

The matching process was designed to identify two potential comparisons for each treatment LEA to 
have a backup, should one comparison LEA refuse to participate. Having a backup LEA reduces the 
risk of losing an entire matched pair if one comparison LEA does not cooperate with on-site data 
collection. 

To obtain the appropriate number of comparisons, a one-to-two match without replacement was 
performed—meaning that once a comparison was selected for one treatment LEA, it could not be 
used for another. To identify pairs for components of data collection, a matching routine was run six 
times, twice for each of the three components. Because there were relatively few LEAs that met the 
stringent Component 3 eligibility requirements, the evaluation team started by matching Component 3 
LEAs to ensure sufficient sample within each component. LEAs were matched in the following order: 

1. Treatment LEAs eligible for Component 3 were matched to the nearest comparison LEA eligible 
for Component 3. 

2. Treatment LEAs eligible for Component 2 and not eligible for Component 3 were matched to the 
nearest comparison LEA eligible for Component 2 that had not been previously selected. 

3. Treatment LEAs eligible for Component 1 and not eligible for Component 2 were matched to the 
nearest comparison LEA eligible for Component 1 that had not been previously selected. 

4. Steps 1, 2, and 3 were repeated, avoided selecting comparison LEAs already selected. 

At each step, to select matches, the evaluation team sorted the treatment LEAs for the component in 
random order and, proceeding from the top of the list, found the comparison LEA eligible for the 
component and nearest in propensity score. If no comparison LEA could be found with a propensity 
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score within 20 percentage points of the treatment LEAs propensity score, the treatment LEA 
remained unmatched. 13 

Since each treatment LEA was included in exactly one matching process during steps 1-3 and exactly 
one during steps 4-6, each treatment LEA was matched with up to two comparison LEAs, which 
were called the first match (from step 1, 2 or 3) and the second match (from step 4, 5 or 6). The first 
match was by construction closer to the treatment in propensity score space. Therefore, the second 
match was used as a backup. 

Some participating LEAs were not matched with any comparisons and were not included in the 
Impact Study sample. Other participating LEAs obtained a first match, but no second match, through 
the matching process. 

2C.1.7 Selecting Schools for On-Site Data Collection 

For outcomes requiring the collection of school-level data, the evaluation team selected samples of 
treatment and comparison schools. The team stratified the schools eligible for on-site data collection 
in the sampled LEAs by grade range (elementary, middle, and high) and selected random samples 
within these strata, giving preference to schools that served a wider grade range. 14 

2C.1.8 Designating a Primary and a Backup Sample 

For the recruiting of the Component 2 and 3 samples, it was necessary to designate a primary sample 
and a back-up sample. To meet this need, the evaluation team generated a uniform random variable 
and sorted the list of treatment LEAs by State, component and this random variable. In order to 
proportionally allocate the targeted 26 Component 2 treatment LEAs and the 26 Component 3 LEAs 
across the States, the appropriate number of treatment LEAs for the primary sample were selected 
from each State according to the randomly generated priority order. Each of these treatment LEAs 
had a comparison LEA designed as the first match, which together comprised the primary comparison 
sample. 

The recruiting team began its work using the primary sample of LEAs. The remaining Component 2 
and 3 LEAs were designated as backups. The backup sample includes two distinct types of 
comparison LEAs: those identified as second matches for treatment LEAs in the primary sample and 
those matched to treatment LEAs in the backup sample. 

2C.2 Propensity Score Modeling Results 

The results from the logistic regressions for CEP participation in these States are summarized in 
Exhibit 2C. 1 below. It is important to note that these models were constructed with the purely 
mechanical objective of capturing (in the independent variables) as much of the variation in 
participation as possible, that is, by maximizing the Pseudo-R squared terms presented in the last row 


13 This algorithm was created for the purpose of this study. Standard algorithms for matching allow for one- 
to-one match without replacement or a one-to-many match with replacement. These would not be suitable 
for this study because one-to-many (in this case, -two) matching without replacement was required. 

14 Within grade range, the evaluation team sorted first by the number of grades served and then by a random 
draw. This insures that a school that serves K-2 is not randomly selected when there’s a more typical 
elementary school serving grades K-5 available. 
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of Exhibit 2C.1. This approach differs from one which seeks to interpret or explain how the 
dependent variable is related to the independent variables. 

The goal of simply capturing as much variation as possible led to the inclusion of some variables and 
the exclusion of others that would be inappropriate if interpretation or explanation were the goal. For 
example, in some cases, variables that are highly multi-collinear: e.g. ISP and percent FRP are two 
measures of poverty, which is highly correlated with urbanicity and FEA size. Multi-collinearity 
poses no threat to maximizing the Pseudo-R squared, but it does render interpretation of coefficients 
problematic. Other examples include certain charter interaction terms, where some variables were 
included in their interacted but not uninteracted form. Because these models were not built with 
interpretive objectives in mind, the evaluation team strongly cautions against using these model 
results to interpret or explain the relationship between particular independent variables and the 
probability of participation. 

Exhibit 2C.1: Primary Data Collection Propensity Score Model Results, by State 



Illinois 

Kentucky 

Michigan 

New York 

Ohio 

ISP 

0.82 

(2.54) 

8.44 

(15.49) 

192.29 

(183.33) 

-10.01 

(19.23) 

4.35 

(31.44) 

Spline 1 (ISP=20-40%) 



-155.92 

(159.54) 



Spline 2 (ISP=40-60%) 



-2.34 

(10.70) 


-2.48 

(9.04) 

Spline 3 (ISP>60%) 



-18.72 

(4.62)*** 


-20.73 

(10.60)* 

% FRP 

6.45 

(1.49)*** 

5.16 

(9.69) 

-13.87 

(13.06) 

16.33 

(11.97) 

6.87 

(19.13) 

(% FRP)/ISP 

-1.04 

(0.48)** 

-0.83 

(3.13) 

4.92 

(4.24) 

-5.09 

(4.89) 

-9.18 

(7.66) 

%ELL 

-6.85 

(2.81)** 

-6.41 

(15.16) 

-7.79 

(3.85)** 

1.10 

(12.29) 

-2.14 

(1.02)** 

%K-5 

4.68 

(2.28)** 

-7.88 

(8.22) 

4.14 

(2.69) 

2.69 

(3.15) 

-3.10 

(0.99)*** 

%6-8 

0.54 

(3.18) 

-0.31 

(16.10) 

-2.09 

(5.84) 

3.78 

(4.79) 

-3.57 

(1.44)** 

City 

-0.62 

(0.91) 

1.03 

(1.35) 

1.66 

(1.03) 

1.51 

(1.33) 

1.00 

(1.09) 

Log (Schools) 

0.91 

(0.37)** 

0.79 

(0.78) 

-0.33 

(0.71) 

2.87 

(1.46)** 


Log (Enrollment) 

-0.28 

(0.28) 

0.23 

(0.68) 

0.55 

(0.43) 

-0.87 

(0.94) 

-0.94 

(0.74) 

Charter 3 



7.14 

(5.14) 

-0.03 

(0.93) 

-37.21 

(18.47)** 

Charter*ISP 



-2.32 

(9.02) 


19.08 

(35.86) 

Charter*%FRP 



3.65 

(9.59) 


-4.34 

(24.59) 

Charter*(%FRP ISP ratio) 



-1.15 

(1.79) 


9.51 

(11.62) 
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Illinois 

Kentucky 

Michigan 

New York 

Ohio 

Charter*ELL% 



7.54 

(4.14)* 



Charter*%K-5 Students 



-4.25 

(2.72) 


3.30 

(1.43)** 

Charter*%6-8 Students 



2.10 

(6.34) 


0.75 

(1.64) 

Charter*City 



-1.31 

(1.21) 


0.96 

(2.03) 

Charter*Log (Schools) 



1.24 

(0.93) 



Charter*Log (Enrollment) 



-0.93 

(0.60) 


1.85 

(0.88)** 

Charter*%Black 



-2.36 

(1.24)* 



Charter*%Hispanic 



0.19 

(2.06) 


0.00 

(0.01) 

Charter*%Asian 



12.96 

(11.78) 



Charter*%Title 1 



0.14 

(1.03) 



Charter*%Magnet 



0.39 

(0.38) 



Constant 

-5.81 

(2.58)** 

-5.26 

(8.82) 

-51.57 

(41.69) 

-1.57 

(9.46) 

13.31 

(11.57) 

N 

424 

169 

623 

638 

721 

Pseudo-R squared 

0.39 

0.28 

0.44 

0.68 

0.40 


a Kentucky and West Virginia have no charter schools. In Illinois, charter schools are members in a regular public 
school LEA and thus part of its food service operations. 

***=p<0.01; **=p<0.05; *=p<0.10 
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2C.3 Recruiting for Onsite Data Collection (Components 2 and 3) 

Once the Impact Study sample was selected, the recruitment of LEAs began. This chapter describes 
the results of those recruitment activities, including the verification of eligibility for the study, and the 
issues with LEAs that arose during recruitment. 

2C.3.1 Recruitment Activities and Results 

Exhibit 2C.2 provides the timeline and activities that were part of the recruiting process. 


Exhibit 2C.2:Timeline of Recruitment Activities 


Activity 

Timing 

Email sent to the FNS Regional Study Liaisons with Study Overview and a 
list of sampled LEAs. 

Oct. 28, 2012 

Email sent to the State Child Nutrition Director with Study Overview and a 
list of sampled LEAs. 

Oct. 30, 2012 

Recruiting packages sent to the primary sample. 

Oct. 31,2012 

Training for recruiters. 

Nov. 6, 2012 

Active recruiting began with calls to LEAs. 

Nov. 7, 2012 

Backup LEAs replaced those LEAs that declined or were found to be 
ineligible. Recruiting packages were sent to replacement LEAs as they 
were selected. 

Nov. 7, 2012 

LEAs agreed to participate. Memorandum of Agreement (MOU) sent to 

LEAs. 

Nov. 7, 2012-Feb. 25, 2013 

Signed MOUs received from LEAs. 

Nov. 8, 2012—April 27, 2013 


In total, 158 LEAs were released for recruitment: the primary sample of 106 LEAs and 52 
replacement LEAs. The final recruitment status of the 158 LEAs was as shown in Exhibit 2C.3. 
During recruitment, one additional pair of LEAs was added to the sample without finding another pair 
ineligible, bringing the number of LEAs actively recruited to 108. 15 


15 An LEA that did not meet the initial eligibility requirement of having three separate cafeterias in the three 
sampled schools was kept in the study because this requirement was causing us to find a number of districts 
ineligible and we became concerned about having enough sample. While deciding whether or not to update 
the eligibility requirements, we kept this match in the sample, and began recruiting a replacement just in 
case. Eligibility requirements were later updated (see Section A.2.c.iii.3) so the LEA that was in question 
was kept in the sample. 
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Exhibit 2C.3: Recruitment Results 


158 LEAs released 


> 


39 LEAs ineligible: 

• 8 treatment 

• 31 comparison 


> 


2 LEAs refused 


> 


9 LEAs dropped: 

• 7 Due to unavailability of matches 

• 2 due to overlap with APEC II Study 
(see Section 2.2.2) 


108 LEAs with MOUs 
signed 


2C.3.2 Verifying Eligibility 

Although the sampling process excluded LEAs that were identified as ineligible based on data in the 
sampling frame, the recruiters took several steps to confirm that the sampled LEAs were still eligible 
for the component(s) for which they were selected. Through this research, the recruiters confirmed 
the following: 

• 70 percent or more of schools in the LEA participated in the CEP (for participating LEAs only) 

• Sampled schools were in operation during the 2012-13 school year 

• Sampled schools were neither alternative nor special needs schools 

• Sampled schools participated in the CEP (for participating LEAs only) 

• LEA schools did not operate a universal free meal program, such as Provision 2 or 3 (for non¬ 
participating LEAs only) 

• The combined enrollment for the sampled schools was 300 students or more (if not, LEA 
enrollment of 300 students or more was also considered eligible) 

• Sampled schools do not share a cafeteria (for Component 3 schools only) 

If any individual school in the LEA did not meet the eligibility criteria, and a backup school was 
available and eligible, the backup school was used instead. 

When recruiters found an LEA to be ineligible, the sampling team confirmed the LEA’s ineligibility 
and provided a replacement LEA and school(s). Exhibit 2C.4 summarizes the final eligibility 
conditions. Because of the nested structure of the data collection, an LEA that was selected for 
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Component 3 is also included in the Implementation Study and Components 1 and 2 of the Impact 
Study. Therefore, it must meet all eligibility requirements for the less intensive Components in 
addition to the Component 3 eligibility requirements. Similarly, a Component 2 LEA must also 
satisfy the eligibility requirements for the Implementation Study and Component 1. A total of 39 
LEAs were found to be ineligible for their selected component. 


Exhibit 2C.4: Final Eligibility Conditions Verified During Recruiting 



Treatment 



LEAs 

Comparison LEAs 

Implementation study eligibility 

LEA in operation during 2012-13 school year 


✓ 

Component 1 eligibility 

LEA schools did not operate a universal free meal program, such 
as Provision 2 or 3 

N/A a 

✓ 

Component 2 eligibility 

70 percent or more of schools in the LEA participated in the CEP 

V 

N/A 

Sampled schools were participating in the CEP 

s 

N/A 

Sampled schools in operation during 2012-13 school year 


✓ 

Sampled schools were neither alternative nor special needs 
schools 

■/ 

✓ 

The combined enrollment for the sampled schools was >300 
students (if not, LEA enrollment of 300 students or more was also 
considered eligible) 


✓ 

Component 3 eligibility 

LEA must have 3 eligible schools 

s 

✓ 

LEA must have at least 2 distinct cafeterias among the sampled 
schools 6 

V 

✓ 

The 3 sampled schools must represent at least two distinct grade 
ranges that that span the K-12 grade range c 

v' 

✓ 


N/A refers to “not applicable.” 

a This eligibility requirement was not verified during the recruiting stage due to the expectation that very few 
LEAs would fall into this category. The PEAR survey screened for participation in alternative universal free meal 
programs. Five LEAs were deemed ineligible for on-site data collection based on their response to this PEAR 
survey screening question and follow up by phone or email: 3 comparison LEAs and 2 treatment LEAs. 

b This is a revision to the original eligibility requirements. The evaluation team’s initial plan required each school 
to have its own cafeteria. 

c This is another revision to the original eligibility requirements. The original plan required three distinct grade 
ranges. 

When recruiters found that an LEA was ineligible for the Component for which it was selected, the 
sampling team investigated its eligibility for less intensive Components of the data collection. LEAs 
found ineligible for Component 3 yet eligible for Component 2 were added to the pool of Component 
2 backup LEAs. LEAs found to be ineligible for Component 1 were removed from the PEAR survey 
sample. 16 LEAs that had closed were removed from the Implementation Study sample. 17 


16 Recruiting information obtained before February 7, 2013 informed the PEAR survey invitations. 

17 Recruiting information obtained before November 30, 2012 informed the Implementation Survey 
invitations. 
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Where possible, the sampling team provided the recruiting team with a replacement LEA. Because 
the recruitment process attempted to preserve the matches from the sampling phase, the recruiting 
team was working with pairs of LEAs. Replacing a treatment LEA required finding a new match for 
the comparison LEA in the pair, and replacing a comparison LEA required finding a new match for 
the treatment LEA in the pair. The LEA with the closest propensity score was selected from among 
the backup LEAs in the State that were eligible for the appropriate Component. 

If no backup LEA was available within the State or if it was not possible to find a new match with a 
propensity score within 20 percentage points of the LEA remaining in the pair, the full match was 
replaced. A new treatment LEA was randomly selected from another State. That LEA and its best 
within-State match were provided to the recruiting team. 

Because the HHFKA requires that States, LEAs, and their contractors participating in the NSLP, 
SBP, and other child nutrition programs cooperate with USDA research and evaluations, the study 
benefitted from high cooperation with data collection. 

Along with this policy, all of the project’s efforts resulted in a very high acceptance rate: only 2 of 
110 LEAs refused to participate, given their eligibility, availability of matches, and cross-study 
participation. This corresponds to a 98 percent recruitment rate. 


Abt Associates 


Appendix 2C |pg.2C-11 





Community Eligibility Provision Evaluation: Final Report 


Appendix 4A: Analysis of Factors Related to Take-Up of the CEP 


This appendix presents supporting information for the discussion in Chapter 4 about the factors 
related to take-up of the CEP by LEAs. The chapter includes results of descriptive analysis of the 
characteristics of participating, eligible non-participating, and near-eligible LEAs and schools, as well 
as determination of factors significantly related to take-up based on regression analysis. The appendix 
begins with information on the data sources and construction of the analysis database. The second 
part of the appendix provides tables of participating, eligible non-participating, and near-eligible LEA 
and school characteristics by State. 

4A.1 Data Sources and Database Construction 

The evaluation team constructed the database for the descriptive analysis of LEA and school 
characteristics (findings are presented in Chapter 4) from several sources. The universe of LEAs for 
the analysis was constructed from the FNS administrative data (Verification Summary Report (VSR) 
data). The States provided data to identify participating LEAs and eligible non-participating LEAs 
from among those listed in the VSR data. States also provided data to identify participating and 
eligible non-participating schools. The evaluation team combined the VSR and State data with LEA 
and school characteristics from the CCD to complete the analysis database. 

While these data sources are the same as those described in Appendix 2A.a for constructing the 
sampling frames of the Matched Administrative Data Sample, the data issues highlighted below are 
specific to the use of these data for the analysis of factors related to take-up of the CEP. The analysis 
of factors related to take-up includes LEAs that are ineligible for inclusion in the Impact Study (e.g. 
private schools) for which missing data are particularly problematic. Discussion of the data sources 
and data issues below is specific to this analysis. 

4A.1.1 Verification Summary Reports 

FNS provided the SY 2011-12 VSR data for the CEP States to the evaluation team, which relied on 
these data for LEA enrollment, the percentage of students eligible for FRP meals, and the ISP used in 
CEP eligibility determination. 1 VSR data were problematic for 15 LEAs (out of 1,388) because, in 
most of these cases, there were extremely large variances between enrollment in the VSR and two 
other reliable sources (the CCD and published State data). Because ISP and FRP calculations relied 
on enrollment, a problem with enrollment meant that the ISP and the FRP percentage were also 
incorrect for these cases. In a few other problematic cases, the computed FRP percentages were 
clearly in error (either zero or 100 percent, and in disagreement with the CCD). Rather than attempt 
to correct the VSR data, all of the cases with problematic enrollment or FRP percentages were 
excluded. Most of the LEAs excluded because of problematic VSR data were small: fifteen had fewer 
than 1,000 students, five had 1,000-5,000 students, two had 5,000-10,000 students, and one had 
10,000-15,000 students. 


See Appendix 2A for a general description of the VSR. Note that all ISP data in the Chapter 4 analysis are 
from the 2011-12 school year—after implementation in the year 1 States and prior to implementation in the 
year 2 States. This represents a key departure from the use of the VSR data in sampling: only pre¬ 
intervention data were used to construct the Sampling Frame or included in the propensity score model. 
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4A.1.2 State Data 

The evaluation team requested data on a number of LEA and school characteristics from State CN 
Agencies. Most importantly, the evaluation team asked States to provide ISP and CEP participation 
data for both LEAs and schools so as to identify CEP eligibility and participation status for LEAs and 
schools. (As explained in Appendix 2A, FNS policy specified that ISP data as of April 1, 2012, 
should be used for determining eligibility for the CEP.) In addition, the evaluation team asked States 
to provide data on LEA and school characteristics for near-eligible, eligible non-participating, and 
participating LEAs and schools to analyze the factors associated with eligibility and participation. 
These additional requested data items included enrollment, percentage of students eligible for free or 
reduced-price lunch, and schools’ grade range and type (e.g., regular, technicaLvocational, special 
education). The States were unable to provide all the elements requested for a variety of reasons. In 
some cases, such as in Kentucky’s Child Nutrition information system, the State did not store school 
enrollment data. In other cases, the data did not exist, as was the case for school ISPs in New York 
and Ohio. 

Exhibit 4A. 1 shows which States provided school-level data. The District of Columbia was the only 
State able to provide all three types of data. New York and Ohio provided no school-level data. The 
other States provided some but not all of the requested data. 


Exhibit 4A.1: School-Level Data Provided by States 



Enrollment 

Identified Student 
Percentage (ISP) 

Free / Reduced Price 
Meals Percentage 

District of Columbia 

Yes 

Yes 

Yes 

Illinois 

Yes 

Partial 3 

Yes 

Kentucky 

No 

Yes 

No 

Michigan 

Yes 

Partial 13 

Yes 

New York 

No 

Partial 0 

No 

Ohio 

No 

Partial 13 

No 

West Virginia 

Yes 

Yes 

No 


a In Illinois, ISP data were missing for 31 percent of schools included in the primary sample for this report. 
b In Michigan, ISP data were missing for 12.5 percent of schools included on the primary sample for this report. 
c In New York, ISP data were missing for 79 percent of schools included on the primary sample for this report. 
d In Ohio, ISP data were missing for 89 percent of schools included on the primary sample for this report. 

4A.1.3 Common Core of Data 

The most recent CCD data available for this report were drawn from SY 2010-11. 2 LEA-level 
variables from the CCD used in this analysis included: the LEA type, the number of traditional 
schools, the number of charter schools, student enrollment by grade, the number of students in Title 1 
schools, the urbanicity of the area in which the LEA is located, and students’ racial and ethnic 
composition. School-level CCD variables used in this analysis were: type of school (e.g., traditional, 
charter, special education), whether the school was Title 1, grade range, and students’ racial and 
ethnic composition. 

The NCES collects biennial data on private schools as part of its Private School Universe Survey 
(PSUS). These data could not be linked to the private schools in the study sample, however, because 


2 See Appendix 2A for an overview of CCD data. 
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the PSUS data do not have an identification numbering system. Moreover, the data are available only 
at the individual private school level, and at the group level (e.g., archdioceses) that can also serve as 
SFAs. 

4A.1.4 Database Construction 

Most LEA-level data summarized in Chapter 4 come from the VSR and CCD. In particular, data on 
enrollment, number of schools, ISP, and the percentage of students eligible for free or reduced-price 
meals come from the VSR. Since VSR data were not available for schools, State data provided 
enrollment, ISP, and the percentage of students eligible for free or reduced-price lunch; when the 
State did not provide one of these data elements, CCD data were used. This section describes how the 
data were combined and key variables were constructed. 

For the descriptive analysis in this report, the evaluation team used the ISP computed from October 
2011 VSR data. The ideal measure would have used the April 2012 State data, since this measure 
determined eligibility for the CEP for SY 2012-13. Elowever, April 2012 data were not available for 
all States and LEAs; there were gaps in the ISP data provided by Illinois, New York, and Ohio. As 
specified by the E1HFKA, the ISP was computed by dividing the number of identified students (free 
meals eligible, not subject to verification) by the total LEA enrollment (excluding students enrolled in 
Provision 2 and 3 schools). 3 

It is possible that the definition used for near-eligible LEAs is less inclusive than it would be with the 
actual April 2012 ISP if that were available for all LEAs in all States. LEAs might have increased 
their ISPs from October 2011 to April 2012. However, the evaluation team used the best data that 
were consistently available. 

The CEP participation and eligibility classification of schools generally follows the classification of 
LEAs. All schools in near-eligible LEAs were classified as near-eligible, and all schools in eligible 
non-participating LEAs were classified as eligible non-participating. However, non-participating 
schools in participating LEAs were classified as follows: (1) eligible non-participating if they either 
belonged to an LEA with an ISP of 40 percent or more (and therefore all schools in the LEA were 
eligible regardless of ISP), had a school ISP of 40 percent or more, or had a missing school ISP; (2) 
near-eligible otherwise (regardless of ISP, because of the possibility of combinations with schools 
with higher ISPs to make a near-eligible group). These definitions were used to ensure consistent 
classifications for all States. The school ISP data needed to construct a more school-specific CEP 
participation and eligibility classification were almost completely missing for two States (New York 
and Ohio) and partially missing for two others (Illinois and Michigan). 

New York City schools were excluded from the schools database for this analysis because most of the 
schools in the list could not be linked to the CCD data. New York City implemented the CEP in 301 
sites. Over two-thirds of the sites in New York City serving school meals under the CEP were special 
education programs that are not identified in the CCD. A smaller number of CEP sites were regular 
schools that previously operated Provision 2 or 3. (The exact number of this latter group of sites was 
not available.) New York City had 1,828 schools; it appears that this total included over 200 schools 
housing special education programs that were treated as separate sites for the CEP. If the New York 


3 Note that this definition differs from the one used in propensity score modeling. See Appendix B for a 
discussion of the ISP used in propensity score modeling. 
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City schools were included, New York would have more total schools than Illinois and more 
participating schools than any other State. 


4A.2 State Tables of LEA and School Characteristics 

Exhibit 4A.2: District of Columbia Regular Public and Charter LEA and School Characteristics 



Near-Eligible 

Eligible Non- 
Partlclpatlng 

Participating 

Combined 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

LEA Variables 





Number of LEAs with VSR data 

5 

12 

19 

36 

Number of LEAs with CCD data 

5 

12 

19 

36 

Enrollment 

457.6 (247.3) 

857.6 (720.3) 

2934.9 (10052.6) 

1898.4 (7307.0) 

Enrollment 1-499 (%) 

60 

41.7 

57.9 

52.8 

Enrollment 500-2499 (%) 

40 

50 

31.6 

38.9 

Enrollment 2500-4999 (%) 

0 

8.3 

5.3 

5.6 

Enrollment 5000+ (%) 

0 

0 

5.3 

2.8 

Percentage distribution of students in 
grades K-12 





Percent in grades K-5 

24.1 (37.5) 

45.4 (35.3) 

58.2 (42.6) 

48.8 (40.2) 

Percent in grades 6-8 

53.4 (37.0) 

19.9 (16.3) 

13.7 (13.7) 

21.8 (23.2) 

Percent in grades 9-12 

22.5 (25.0) 

34.7 (38.3) 

28.1 (38.7) 

29.5 (36.1) 

Number of schools 

1.2 (0.4) 

2.9 (2.7) 

8.1 (27.6) 

5.4 (20.1) 

1 school (%) 

80 

50 

68.4 

63.9 

2-5 schools (%) 

20 

33.3 

15.8 

22.2 

6-14 schools (%) 

0 

16.7 

10.5 

11.1 

15+ schools (%) 

0 

0 

5.3 

2.8 

Identified Student Percentage (ISP) 

37.4 (5.4) 

39.1 (9.5) 

46.6 (14.3) 

42.9 (12.4) 

Students free/reduced lunch (%) 

68.4 (8.5) 

80.4 (6.3) 

82.4 (12.4) 

79.8 (11.1) 

Any charter schools (%) 

100 

91.7 

94.1 

94.1 

Percent Title 1 schools 

100 (0.0) 

86.5 (30.8) 

94.2 (13.9) 

92.3 (20.8) 

Urban LEA (%) 

100 

100 

100 

100 

Percent students Black 

90 (8.5) 

92.7 (9.6) 

94.7 (10.1) 

93.3 (9.6) 

Percent students Hispanic/Latino 

7.7 (6.8) 

5 (7.2) 

3.9 (8.5) 

4.9 (7.7) 

School Variables 





Number of schools with enrollment data 

39 

35 

116 

190 

Number of schools with ISP 

39 

35 

116 

190 

Number of schools with FRP percentage 

35 

30 

112 

177 

Number of schools with CCD data 

35 

30 

114 

179 

Enrollment 

448.2 (296.9) 

289.9 (176.2) 

349.6 (170.8) 

358.8 (208.9) 

ISP 

20.3 (13.6) 

43 (10.7) 

60.1 (13.5) 

48.8 (20.5) 

Students free/reduced lunch (%) 

52.6 (34.8) 

81.6 (10.3) 

83.2 (22.4) 

76.9 (26.7) 

Grade span 





K-5 (%) 

51.4 

53.3 

46.5 

48.6 

6-8 (%) 

14.3 

20 

11.4 

13.4 

9-12 (%) 

17.1 

13.3 

13.2 

14 

Other (%) 

17.1 

13.3 

27.2 

22.9 

Title 1 schools (%) 

62.9 

83.3 

96.5 

87.7 

Charter schools (%) 

17.1 

96.7 

28.1 

37.4 

Percent students Black 

54.9 (31.2) 

89.1 (16.8) 

88.9 (19.1) 

82.2 (25.5) 

Percent students Hispanic/Latino 

13.8 (17.8) 

8.3 (14.4) 

9.4 (17.5) 

10.1 (17.1) 


LEAs in the District of Columbia have either all charter or no charter schools. 
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Exhibit 4A.3: Illinois Regular Public and Charter LEA and School Characteristics 



Near-Eligible 

Eligible Non- 
Participating 

Participating 

Combined 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

LEA Variables 





Number of LEAs with VSR data 

89 

103 

48 

240 

Number of LEAs with CCD data 

93 

104 

48 

245 

Enrollment 

1126.4 (1283.9) 

3541.9 (5902.5) 

9901.4 (54514.4) 

3918.1 (24699.1) 

Enrollment 1-499 (%) 

43.8 

24.3 

31.3 

32.9 

Enrollment 500-2499 (%) 

46.1 

42.7 

43.8 

44.2 

Enrollment 2500-4999 (%) 

6.7 

13.6 

12.5 

10.8 

Enrollment 5000+ (%) 

3.4 

19.4 

12.5 

12.1 

Percentage distribution of students in 
grades K-12 





Percent in grades K-5 

53.5 (18.6) 

54.2 (11.8) 

59.3 (10.8) 

55 (14.7) 

Percent in grades 6-8 

27.3 (9.9) 

26.6 (6.1) 

28.7 (6.0) 

27.3 (7.7) 

Percent in grades 9-12 

19.2 (28.0) 

19.3 (17.3) 

12 (15.0) 

17.7 (21.7) 

Number of schools 

2.9 (2.3) 

7 (8.4) 

17.4 (83.3) 

7.6 (37.8) 

1 school (%) 

33.7 

13.6 

22.9 

22.9 

2-5 schools (%) 

56.2 

50.5 

45.8 

51.7 

6-14 schools (%) 

10.1 

24.3 

22.9 

18.8 

15+ schools (%) 

0 

11.7 

8.3 

6.7 

Identified Student Percentage (ISP) 

34.7 (8.8) 

40 (18.1) 

51.4 (20.2) 

40.3 (16.9) 

Students free/reduced lunch (%) 

63.1 (15.4) 

66 (17.9) 

62.9 (21.2) 

64.3 (17.7) 

Any charter schools (%) 

1.1 

4.1 

8.3 

3.8 

Percent Title 1 schools 

92.7 (20.7) 

88.3 (18.4) 

94.8 (13.0) 

91.3 (18.5) 

Urban LEA (%) 

2.3 

11.2 

6.3 

6.8 

Percent students Black 

12.2 (21.8) 

16.8 (25.5) 

38.6 (36.6) 

19.6 (28.6) 

Percent students Hispanic/Latino 

11.5 (17.2) 

15.6 (21.6) 

11.1 (17.3) 

13.1 (19.2) 

School Variables 





Number of schools with enrollment data 

272 

1109 

444 

1825 

Number of schools with ISP 

73 

734 

449 

1256 

Number of schools with FRP percentage 

229 

1026 

414 

1669 

Number of schools with CCD data 

229 

1043 

416 

1688 

Enrollment 

388.6 (401.1) 

580.1 (480.8) 

423.7 (231.5) 

513.5 (429.3) 

ISP 

35.2 (2.9) 

46.2 (11.7) 

61.8 (20.1) 

51.1 (17.2) 

Students free/reduced lunch (%) 

58.8 (17.7) 

65.8 (24.1) 

88.3 (14.9) 

70.4 (23.8) 

Grade span 





K-5 (%) 

49.8 

47.8 

30 

43.7 

6-8 (%) 

23.6 

13.9 

10.3 

14.3 

9-12 (%) 

14.8 

12.2 

5.8 

11 

Other (%) 

11.8 

26.1 

53.8 

31 

Title 1 schools (%) 

94.3 

81.4 

98.6 

87.4 

Charter schools (%) 

0 

1.8 

0.5 

1.2 

Percent students Black 

15.6 (25.1) 

21.9 (27.9) 

69.2 (36.3) 

32.7 (36.5) 

Percent students Hispanic/Latino 

15.4 (21.1) 

32.4 (32.5) 

13.8 (26.2) 

25.5 (31.0) 


Note: LEAs in Illinois mix charter and non-charter schools. 
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Exhibit 4A.4: Kentucky Regular Public and Charter LEA and School Characteristics 



Near-Eligible 

Eligible Non- 
Participating 

Participating 

Combined 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

LEA Variables 





Number of LEAs with VSR data 

12 

92 

51 

155 

Number of LEAs with CCD data 

12 

92 

51 

155 

Enrollment 

1771.1 (1434.1) 

4909.1 (11047.2) 

3440.3 (2921.6) 

4182.8 (8716.9) 

Enrollment 1-499 (%) 

16.7 

5.4 

2 

5.2 

Enrollment 500-2499 (%) 

50 

44.6 

47.1 

45.8 

Enrollment 2500-4999 (%) 

25 

31.5 

35.3 

32.3 

Enrollment 5000+ (%) 

8.3 

18.5 

15.7 

16.8 

Percentage distribution of students in 
grades K-12 





Percent in grades K-5 

47.2 (7.2) 

47.4 (4.6) 

47.2 (2.6) 

47.3 (4.3) 

Percent in grades 6-8 

24.7 (2.7) 

22.8 (1.3) 

22.8 (1.2) 

22.9 (1.5) 

Percent in grades 9-12 

28.2 (9.1) 

29.8 (5.0) 

30.1 (2.3) 

29.8 (4.8) 

Number of schools 

3.3 (2.6) 

8.9 (16.2) 

9.5 (13.9) 

8.6 (14.9) 

1 school (%) 

25 

4.3 

0 

4.5 

2-5 schools (%) 

66.7 

40.2 

41.2 

42.6 

6-14 schools (%) 

8.3 

45.7 

47.1 

43.2 

15+ schools (%) 

0 

9.8 

11.8 

9.7 

Identified Student Percentage (ISP) 

29 (4.5) 

32.2 (8.7) 

45.9 (16.4) 

36.5 (13.4) 

Students free/reduced lunch (%) 

61.9 (7.9) 

59 (11.7) 

63.5 (13.0) 

60.7 (12.0) 

Any charter schools (%) 

0 

0 

0 

0 

Percent Title 1 schools 

82.7 (20.2) 

77.1 (19.6) 

76.9 (15.1) 

77.4 (18.2) 

Urban LEA (%) 

0 

5.4 

3.9 

4.5 

Percent students Black 

3.5 (7.2) 

5.7 (7.4) 

6.8 (10.6) 

5.9 (8.5) 

Percent students Hispanic/Latino 

2.1 (1.3) 

3.3 (3.0) 

2.3 (3.7) 

2.9 (3.2) 

School Variables 





Number of schools with enrollment data 

107 

812 

246 

1165 

Number of schools with ISP 

110 

855 

268 

1233 

Number of schools with FRP percentage 

107 

809 

246 

1162 

Number of schools with CCD data 

108 

815 

249 

1172 

Enrollment 

608.3 (273.0) 

548.7 (336.4) 

402.4 (210.0) 

523.3 (314.9) 

ISP 

27.7 (9.6) 

39.3 (16.0) 

56.9 (12.0) 

42.1 (17.0) 

Students free/reduced lunch (%) 

52.7 (14.2) 

60.5 (19.5) 

73.7 (14.0) 

62.5 (19.0) 

Grade span 





K-5 (%) 

41.7 

56.8 

51.4 

54.3 

6-8 (%) 

24.1 

18.7 

8.8 

17.1 

9-12 (%) 

25.9 

16.1 

13.3 

16.4 

Other (%) 

8.3 

8.3 

26.5 

12.2 

Title 1 schools (%) 

89.8 

83.1 

91.6 

85.5 

Charter schools (%) 

0 

0 

0 

0 

Percent students Black 

5.8 (7.2) 

12.8 (17.0) 

5.5 (11.3) 

10.6 (15.6) 

Percent students Hispanic/Latino 

2.7 (2.1) 

4.8 (6.4) 

1.8 (3.6) 

4 (5.7) 


pg.4A-6| Appendix 4A 


Abt Associates 



















































Community Eligibility Provision Evaluation: Final Report 


Exhibit 4A.5: Michigan Regular Public and Charter LEA and School Characteristics 



Near-Eligible 

Eligible Non- 
Participating 

Participating 

Combined 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

LEA Variables 





Number of LEAs with VSR data 

20 

353 

107 

480 

Number of LEAs with CCD data 

20 

358 

111 

489 

Enrollment 

900.7 (997.3) 

2276.8 (3149.7) 

2677.9 (7692.8) 

2308.1 (4523.1) 

Enrollment 1-499 (%) 

55 

20.4 

35.8 

25.3 

Enrollment 500-2499 (%) 

40 

52.4 

41.5 

49.5 

Enrollment 2500-4999 (%) 

5 

17.3 

12.3 

15.7 

Enrollment 5000+ (%) 

0 

9.9 

10.4 

9.6 

Percentage distribution of students in 
grades K-12 





Percent in grades K-5 

52.6 (16.2) 

46.6 (14.7) 

52.3 (18.5) 

48.1 (15.8) 

Percent in grades 6-8 

22.9 (3.8) 

23.4 (6.5) 

22.1 (7.1) 

23.1 (6.6) 

Percent in grades 9-12 

24.5 (15.3) 

30 (16.3) 

25.6 (18.6) 

28.8 (16.9) 

Number of schools 

2.1 (1.6) 

4.7 (5.0) 

6.1 (14.2) 

4.9 (8.0) 

1 school (%) 

60 

26.3 

41.1 

31 

2-5 schools (%) 

35 

44.2 

31.8 

41 

6-14 schools (%) 

5 

24.9 

19.6 

22.9 

15+ schools (%) 

0 

4.5 

7.5 

5 

Identified Student Percentage (ISP) 

31.4 (7.0) 

33.3 (13.2) 

56.1 (20.0) 

38.3 (17.6) 

Students free/reduced lunch (%) 

55.7 (12.1) 

60.5 (18.3) 

69.1 (19.7) 

62.2 (18.8) 

Any charter schools (%) 

30 

24.2 

40.8 

28 

Percent Title 1 schools 

91.1 (15.3) 

85.2 (19.6) 

87.3 (17.6) 

85.9 (19.1) 

Urban LEA (%) 

10 

16.6 

37.9 

20.9 

Percent students Black 

6.9 (21.8) 

21 (32.7) 

43.3 (39.6) 

25.2 (35.3) 

Percent students Hispanic/Latino 

7.4 (10.8) 

5.7 (9.6) 

7.5 (11.0) 

6.2 (10.0) 

School Variables 





Number of schools with enrollment data 

104 

1897 

512 

2513 

Number of schools with ISP 

104 

1897 

512 

2513 

Number of schools with FRP percentage 

100 

1801 

461 

2362 

Number of schools with CCD data 

108 

2210 

475 

2793 

Enrollment 

505 (362.4) 

440.1 (332.4) 

433.2 (256.8) 

441.4 (319.9) 

ISP 

27.3 (8.8) 

37.2 (16.7) 

66.2 (12.5) 

42.7 (19.7) 

Students free/reduced lunch (%) 

43.5 (13.9) 

51.6 (21.4) 

81 (13.2) 

57 (23.1) 

Grade span 





K-5 (%) 

45.4 

46.8 

48.8 

47.1 

6-8 (%) 

16.7 

16.3 

12.2 

15.6 

9-12 (%) 

24.1 

23.8 

15.8 

22.4 

Other (%) 

13.9 

11.8 

22.5 

13.7 

Title 1 schools (%) 

69.4 

70.6 

93.9 

74.5 

Charter schools (%) 

5.6 

5.2 

11.8 

6.4 

Percent students Black 

4 (11.5) 

17.1 (27.2) 

53.8 (37.4) 

23.5 (32.6) 

Percent students Hispanic/Latino 

5.2 (6.8) 

6.1 (10.5) 

11 (17.5) 

7 (12.2) 


LEAs in Michigan have either all charter or no charter schools. 
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Exhibit 4A.6: New York Regular Public and Charter LEA and School Characteristics 



Near-Eligible 

Eligible Non- 
Participating 

Participating 

Combined 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

LEA Variables 





Number of LEAs with VSR data 

48 

12 

17 

77 

Number of LEAs with CCD data 

48 

12 

17 

77 

Enrollment 

1754.3 (2470.2) 

2403.3 (2824.4) 

68529 

16598 

Enrollment 1-499 (%) 

39.6 

33.3 

29.4 

36.4 

Enrollment 500-2499 (%) 

41.7 

50 

11.8 

36.4 

Enrollment 2500-4999 (%) 

10.4 

0 

5.9 

7.8 

Enrollment 5000+ (%) 

8.3 

16.7 

52.9 

19.5 

Percentage distribution of students in 
grades K-12 





Percent in grades K-5 

50.3 (25.5) 

61.1 (23.0) 

49.5 (17.3) 

51.9 (23.7) 

Percent in grades 6-8 

24.3 (18.5) 

17.5 (9.7) 

30.5 (21.4) 

24.5 (18.2) 

Percent in grades 9-12 

25.4 (23.8) 

21.4 (13.9) 

20 (14.7) 

23.6 (20.7) 

Number of schools 

3.4 (4.2) 

4.3 (3.7) 

122.9 (440.0) 

29.9 (208.0) 

1 school (%) 

50 

25 

35.3 

42.9 

2-5 schools (%) 

31.3 

58.3 

5.9 

29.9 

6-14 schools (%) 

14.6 

16.7 

29.4 

18.2 

15+ schools (%) 

4.2 

0 

29.4 

9.1 

Identified Student Percentage (ISP) 

39.4 (15.0) 

32.5 (18.5) 

41.6 (22.2) 

38.8 (17.3) 

Students free/reduced lunch (%) 

70.2 (17.6) 

62.6 (21.4) 

79.8 (12.9) 

71.1 (17.9) 

Any charter schools (%) 

40 

41.7 

35.7 

39.4 

Percent Title 1 schools 

87.3 (29.4) 

90.9 (15.8) 

77.1 (33.9) 

85.9 (28.6) 

Urban LEA (%) 

48.9 

41.7 

86.7 

55.6 

Percent students Black 

38.9 (35.6) 

33.7 (30.4) 

57.5 (25.0) 

41.7 (33.6) 

Percent students Hispanic/Latino 

12.6 (15.8) 

19.7 (19.5) 

15.5 (14.0) 

14.4 (16.1) 

School Variables 





Number of schools with enrollment data 

158 

106 

163 

427 

Number of schools with ISP 

0 

0 

102 

102 

Number of schools with FRP percentage 

158 

105 

163 

426 

Number of schools with CCD data 

159 

110 

164 

433 

Enrollment 

511.6 (343.9) 

549.7 (503.0) 

586.1 (284.5) 

549.5 (371.2) 

ISP 



59.7 (23.2) 

59.7 (23.2) 

Students free/reduced lunch (%) 

64.3 (17.3) 

63 (19.6) 

80.5 (13.5) 

70.2 (18.5) 

Grade span 





K-5 (%) 

56.6 

46.4 

51.2 

52 

6-8 (%) 

15.7 

21.8 

8.5 

14.5 

9-12 (%) 

14.5 

16.4 

11 

13.6 

Other (%) 

13.2 

13.6 

28.7 

19.2 

Title 1 schools (%) 

92.5 

80 

91.5 

88.9 

Charter schools (%) 

11.3 

4.5 

3 

6.5 

Percent students Black 

30.3 (29.4) 

40.1 (27.3) 

53.8 (23.4) 

41.7 (28.5) 

Percent students Hispanic/Latino 

12.3 (13.5) 

16.7 (14.1) 

18.2 (17.2) 

15.6 (15.4) 


LEAs in New York have either all charter or no charter schools. 
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Exhibit 4A.7: Ohio Regular Public and Charter LEA and School Characteristics 



Near-Eligible 

Eligible Non- 
Participating 

Participating 

Combined 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

LEA Variables 





Number of LEAs with VSR data 

101 

146 

68 

315 

Number of LEAs with CCD data 

102 

146 

68 

316 

Enrollment 

1680.8 (1563.0) 

1687.6 (5470.8) 

1988.1 (6141.0) 

1750.3 (4760.4) 

Enrollment 1-499 (%) 

26.7 

61 

66.2 

51.1 

Enrollment 500-2499 (%) 

50.5 

27.4 

22.1 

33.7 

Enrollment 2500-4999 (%) 

16.8 

7.5 

2.9 

9.5 

Enrollment 5000+ (%) 

5.9 

4.1 

8.8 

5.7 

Percentage distribution of students in 
grades K-12 





Percent in grades K-5 

49.2 (20.8) 

53.6 (28.0) 

53.3 (31.7) 

52 (26.5) 

Percent in grades 6-8 

22.8 (11.2) 

23 (18.1) 

22.5 (21.3) 

22.8 (16.7) 

Percent in grades 9-12 

28 (21.1) 

23.4 (31.0) 

24.2 (31.8) 

25.2 (28.1) 

Number of schools 

3.4 (2.9) 

3.7 (11.2) 

4.6 (13.0) 

3.8 (9.9) 

1 school (%) 

37.6 

68.5 

72.1 

59.4 

2-5 schools (%) 

45.5 

19.2 

16.2 

27 

6-14 schools (%) 

16.8 

8.9 

4.4 

10.5 

15+ schools (%) 

0 

3.4 

7.4 

3.2 

Identified Student Percentage (ISP) 

34.3 (9.5) 

50.9 (13.7) 

53.1 (17.2) 

46 (15.6) 

Students free/reduced lunch (%) 

65 (15.0) 

82.8 (13.6) 

83.8 (14.4) 

77.3 (16.5) 

Any charter schools (%) 

27.7 

63.2 

68.3 

51.7 

Percent Title 1 schools 

95.7 (11.6) 

97.3 (8.0) 

98.2 (7.3) 

96.9 (9.3) 

Urban LEA (%) 

26.7 

57.6 

63.3 

47.9 

Percent students Black 

21.7 (30.7) 

49.5 (36.5) 

43.1 (36.3) 

38.4 (36.6) 

Percent students Hispanic/Latino 

4.1 (11.4) 

4.7 (7.9) 

6.8 (10.5) 

4.9 (9.8) 

School Variables 





Number of schools with enrollment data 

378 

586 

295 

1259 

Number of schools with ISP 

0 

2 

151 

153 

Number of schools with FRP percentage 

378 

585 

293 

1256 

Number of schools with CCD data 

386 

627 

314 

1327 

Enrollment 

451.8 (270.0) 

441 (264.6) 

405.9 (254.0) 

436 (264.2) 

ISP 


70.8 (8.4) 

61.6 (15.3) 

61.7 (15.2) 

Students free/reduced lunch (%) 

56.3 (25.7) 

72 (27.1) 

76.5 (24.6) 

68.3 (27.3) 

Grade span 





K-5 (%) 

52.3 

44 

30.9 

43.3 

6-8 (%) 

21.2 

15.3 

8.6 

15.4 

9-12 (%) 

17.4 

17.2 

15.6 

16.9 

Other (%) 

8.5 

22.8 

43.6 

23.6 

Title 1 schools (%) 

94 

88.5 

92 

91 

Charter schools (%) 

7.3 

14.7 

14 

12.4 

Percent students Black 

14.8 (24.0) 

42.4 (33.6) 

52.4 (34.8) 

36.4 (34.6) 

Percent students Hispanic/Latino 

3.1 (7.0) 

5.3 (8.0) 

7.1 (13.0) 

5.1 (9.2) 


LEAs in Ohio have either all charter or no charter schools. 
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Exhibit 4A.8: West Virginia Regular Public and Charter LEA and School Characteristics 



Near-Eligible 

Eligible Non- 
Participating 

Participating 

Combined 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

LEA Variables 





Number of LEAs with VSR data 

1 

19 

35 

55 

Number of LEAs with CCD data 

1 

19 

35 

55 

Enrollment 

1080 

4897.4 (4271.8) 

5417.9 (5400.8) 

5159.2 (4982.3) 

Enrollment 1-499 (%) 

0 

0 

0 

0 

Enrollment 500-2499 (%) 

100 

42.1 

28.6 

34.5 

Enrollment 2500-4999 (%) 

0 

31.6 

37.1 

34.5 

Enrollment 5000+ (%) 

0 

26.3 

34.3 

30.9 

Percentage distribution of students in 
grades K-12 





Percent in grades K-5 

45.2 

45.8 (1.5) 

46.2 (1.9) 

46 (1.7) 

Percent in grades 6-8 

22.5 

23.1 (0.7) 

23.1 (0.9) 

23.1 (0.8) 

Percent in grades 9-12 

32.3 

31.1 (1.6) 

30.7 (1.8) 

30.9 (1.8) 

Number of schools 

3 

11.1 (8.3) 

13.8 (12.0) 

12.7 (10.8) 

1 school (%) 

0 

0 

0 

0 

2-5 schools (%) 

100 

31.6 

17.1 

23.6 

6-14 schools (%) 

0 

42.1 

48.6 

45.5 

15+ schools (%) 

0 

26.3 

34.3 

30.9 

Identified Student Percentage (ISP) 

26.9 

34 (5.4) 

38.1 (7.2) 

36.5 (7.0) 

Students free/reduced lunch (%) 

57 

52.2 (7.4) 

55.8 (8.4) 

54.5 (8.1) 

Any charter schools (%) 

0 

0 

0 

0 

Percent Title 1 schools 

66.7 

50.7 (17.2) 

56 (15.8) 

54.4 (16.3) 

Urban LEA (%) 

0 

15.8 

5.7 

9.1 

Percent students Black 

1 

2.3 (2.3) 

3.3 (3.8) 

2.9 (3.3) 

Percent students Hispanic/Latino 

0.3 

0.8 (0.9) 

0.8 (1.2) 

0.8 (1.1) 

School Variables 





Number of schools with enrollment data 

163 

248 

284 

695 

Number of schools with ISP 

163 

248 

284 

695 

Number of schools with FRP percentage 

160 

248 

279 

687 

Number of schools with CCD data 

162 

248 

280 

690 

Enrollment 

521.7 (342.2) 

421.9 (261.0) 

317.6 (180.9) 

402.7 (267.1) 

ISP 

29.7 (6.8) 

38.6 (12.4) 

49.3 (12.0) 

40.9 (13.6) 

Students free/reduced lunch (%) 

44.7 (10.4) 

51.9 (13.3) 

63.7 (12.9) 

55 (14.6) 

Grade span 





K-5 (%) 

38.9 

60.5 

70.4 

59.4 

6-8 (%) 

27.2 

17.3 

11.4 

17.2 

9-12 (%) 

24.1 

14.9 

6.4 

13.6 

Other (%) 

9.9 

7.3 

11.8 

9.7 

Title 1 schools (%) 

20.4 

50.8 

70.4 

51.6 

Charter schools (%) 

0 

0 

0 

0 

Percent students Black 

4.2 (5.7) 

3.1 (5.7) 

6.4 (10.9) 

4.7 (8.4) 

Percent students Hispanic/Latino 

1.3 (2.0) 

0.8 (1.2) 

0.9 (1.9) 

1 (1.7) 
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Exhibit 4A.9: Non-Public, Non Regular, and Non Charter LEAs—LEA and School 
Characteristics 



Near-Eligible 

Eligible Non- 
Participating 

Participating 

Combined 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

LEA Variables 





Number of LEAs with VSR data 

157 

82 

71 

310 

Number of LEAs with CCD data 

157 

90 

71 

318 

Enrollment 

180.4 (268.6) 

1288.6 (6768.4) 

765.3 (1470.8) 

607.5 (3571.7) 

Enrollment 1-499 (%) 

92.4 

91.5 

70.4 

87.1 

Enrollment 500-2499 (%) 

7.6 

3.7 

21.1 

9.7 

Enrollment 2500-4999 (%) 

0 

0 

5.6 

1.3 

Enrollment 5000+ (%) 

0 

4.9 

2.8 

1.9 

Percentage distribution of students in 
grades K-12 





Percent in grades K-5 


15.6 (18.2) 

25.6 

16.3 (17.7) 

Percent in grades 6-8 


23.7 (29.4) 

25.2 

23.8 (28.2) 

Percent in grades 9-12 


60.7 (34.7) 

49.1 

59.8 (33.4) 

Number of schools 

2.7 (3.9) 

5.2 (23.0) 

2 (1.9) 

3.2 (12.2) 

1 school (%) 

58 

68.3 

59.2 

61 

2-5 schools (%) 

31.2 

25.6 

35.2 

30.6 

6-14 schools (%) 

8.9 

2.4 

5.6 

6.5 

15+ schools (%) 

1.9 

3.7 

0 

1.9 

Identified Student Percentage (ISP) 

65.6 (27.6) 

53.6 (22.6) 

28.1 (29.5) 

53.8 (30.6) 

Students free/reduced lunch (%) 

84.2 (16.8) 

82.7 (18.4) 

82.7 (17.4) 

83.4 (17.3) 

Any charter schools (%) 

0 

0 

0 

0 

Percent Title 1 schools 

0 (0.0) 

0 (0.0) 

O 

o 

0 (0.0) 

Urban LEA (%) 

9.1 

7.5 

30 

10.8 

Percent students Black 

13 

12.2 (11.5) 

14.9 (12.6) 

12.7 (11.0) 

Percent students Hispanic/Latino 

7.4 

3.1 (4.0) 

7.6 (10.3) 

4.1 (5.3) 

School Variables 





Number of schools with enrollment data 

78 

306 

20 

404 

Number of schools with ISP 

20 

191 

81 

292 

Number of schools with FRP percentage 

16 

42 

6 

64 

Number of schools with CCD data 

36 

118 

12 

166 

Enrollment 

49.7 (58.5) 

219.9 (225.4) 

89.8 (83.0) 

180.2 (210.0) 

ISP 

35.8 (2.8) 

48.5 (30.8) 

62.7 (15.4) 

51.9 (27.2) 

Students free/reduced lunch (%) 

68.6 (29.8) 

59.4 (25.8) 

61 (33.9) 

61.8 (27.8) 

Grade span 





K-5 (%) 

2.9 

7.5 

8 

6.6 

6-8 (%) 

0 

5.7 

0 

3.6 

9-12 (%) 

17.1 

18.9 

20 

18.7 

Other (%) 

68.6 

42.5 

44 

48.2 

Title 1 schools (%) 

0 

0 

0 

0 

Charter schools (%) 

0 

0 

0 

0 

Percent students Black 

13.1 (19.0) 

17.7 (19.4) 

19.3 (25.6) 

16.9 (20.2) 

Percent students Hispanic/Latino 

4.4 (6.9) 

3 (5.2) 

7.7 (17.0) 

4 (8.4) 
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Exhibit 4A.10: Median Ratio of FRP Student Percentage to ISP for Eligible Non-Participating, 
Participating, and All Eligible Regular LEAs, by State 



Eligible 

Non- 

Participating 

Participating 

All Eligible 
LEAs 

District of Columbia 

2.07 

1.76 

1.90 

Illinois 

2.00 

1.67 

1.87 

Kentucky 

1.95 

1.75 

1.87 

Michigan 

1.99 

1.80 

1.97 

New York 

1.79 

1.52 

1.66 

Ohio 

1.60 

1.49 

1.56 

West Virginia 

1.52 

1.48 

1.49 

Total 

1.90 

1.65 

1.84 


Sources: Sources: FNS Verification Summary (FNS-742) data, State lists of eligible and participating LEAs. 
Projected ISP based on October 2011 VSR data. The definition of an eligible LEA varies by State, as explained 
in Exhibit 2.1. 
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Appendix 5A: Analysis of Participation and Revenue 


For estimating the impacts of the CEP on participation and revenues, a comparative interrupted time 
series (CITS) analysis was conducted with a baseline linear trend projection model. The following 
discussion explains and justifies the projection model selected and describes the CITS method used. 


5A.1 Analysis of Participation 

5A.1.1 Baseline Balance Testing for Participation 

The NSLP participation sample is identical to the matched administrative sample, for which Chapter 
2 reported the results of balance testing. However, the matched sample for SBP differs due to some 
missing information. Exhibit 5 A. 1 shows the results of baseline balance testing for the SBP 
participation sample. 


Exhibit 5A.1: SBP Participation Sample Characteristics, Weighted to Reflect Many to One 
Matching, LEA Level Measures 



Hypothesis testing performed using t-tests for all outcomes except for grade distribution. 
Grade distribution outcomes jointly tested using MANOVA. 

***=p<.01, **=p<05, *= p<.10 

Source: Common Core of School Data, State-Provided Data on SBP and NSLP Participation 
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5A.1.2 Selection of Baseline Projection Model for Participation Impact 

The first step in the specification of the analysis was the selection of a baseline projection model. The 
evaluation team began by plotting the year-to-year pattern of the mean of average daily participation 
(ADP) for each State separately and for all States combined. These plots were constructed using the 
unadjusted ADP for each LEA aggregated over States. Exhibits 5A.2 and 5A.3 present these graphs 
for all States combined. Exhibit 5A.2 depicts the trend in participation in the National School Lunch 
Program (NSLP), and Exhibit 5A.3 depicts the trend in participation in the School Breakfast Program 
(SBP). Points in the graphs represent the mean ADP for each year measured. As the figures suggest, 
the ADP values varied from SY 2009-10 and 2010-11. The rate also exhibits a clear and systematic 
upward trend. 

Baseline graphs for individual States were inspected for systematic trends that might be masked by 
the combined findings in Exhibits 5A.2 and 5A.3. Individual States had considerably more year-to- 
year fluctuations in their mean ADP than was suggested by the combined findings. (Exhibits are not 
shown but are available on request.) However, they also exhibited a clear and systematic upward 
trend in the pre-intervention years. On balance, then, visual inspection of the baseline ADP for the 
States, both combined and individually, indicated that a baseline trend model was appropriate for 
estimating the impacts of the CEP. Corresponding evidence from a statistical analysis of baseline data 
supports the same conclusion. This evidence was obtained by fitting a regression adjusted linear trend 
model to the pre-intervention ADP for LEAs. Results of that analysis are available on request. 

Exhibit 5A.2: Average Daily Participation in NSLP for Treatment and Comparison LEAs over 
Time 


ADP for Treatment and Comparison LEAs over time 
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Exhibit 5A.3: Average Daily Participation in SBP for Treatment and Comparison LEAs over 
Time 


ADP for Treatment and Comparison LEAs over time 
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5A.1.3 Visual Depiction of Participation Results 

As a visual guide for interpreting the CITS impact estimates of CEP on participation in NSLP and the 
SBP, Exhibits 5A.4 and 5A.5 display the regression-adjusted ADP over time for both the treatment 
and comparison LEAs. In this example, there are two pre-treatment years (2009 and 2010) and two 
post-treatment years (2011 and 2012). The solid lines represent the observed fitted regression- 
adjusted ADP (blue for treatment LEAs and red for comparison LEAs). 1 The dashed line indicates the 
estimate of the ADP in the absence of the intervention (baseline linear projection—blue for the 
treatment LEAs and red for the comparison LEAs). 2 The observed ADP (represented by the solid blue 
and red lines) differs from the projected ADP in the post-implementation years (the counterfactual 
represented by the dashed blue and red lines), and it is these differences that allow us to net out the 
effects of other contextual or maturation processes. The difference for the treatment group minus the 
difference for the comparison group equals the estimated impact. 

For the NSLP, the average difference for the treatment group was four percent, while the average 
difference for the comparison group was 0.4 percent, resulting in a difference-in-difference estimate 
of 3.5 percent. 3 For the SBP, the average difference for the treatment group is 6 percent, and the 
average difference for the comparison group is 2.4 percent, and the estimated impact was 3.6 percent. 


1 For the purposes of the exhibits, the annual means are for Year 1 States (LEAs that adopted the intervention 
in SY 2011-12 and the comparison group in those States). The pre- and post-intervention time periods for 
LEAs that implemented the CEP in 2012 are different, and so the regression-adjusted means may be 
different, but the estimated impact is the same because it derives from the same parameter in the model. 

In the model notation shown in Chapter 5, the intercept for the comparison LEAs is the parameter estimate 
for (ft), and the slope is(ft) ■ The intercept for treatment LEAs is the estimate (ft + ft) and the slope is 
the same by design (ft). 

The estimated impact of 3.5 percent differs from the computed difference between the treatment and 
control group differences (4.0 percent -0.4 percent) due to rounding. 
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Exhibit 5A.4: Impact on NSLP Participation Rate 


Impact on NSLP participation rate 



Treatment —•- Projected Treatment 

Comparison —•- Projected Comparison 


Exhibit 5A.5: Impact on SBP Participation Rate 


Impact on SBP participation rate 



Treatment —•- Projected Treatment 

Comparison —•- Projected Comparison 


Abt Associates 


Appendix 5A | pg. 5A-5 






























Community Eligibility Provision Evaluation: Final Report 


5A.1.4 Regression Results for Participation 

While Chapter 5 specifically reports the estimated impacts, this Appendix exhibits presents all of the 
parameter estimates from the models used to estimate impacts on NSLP and SBP participation. The 
estimated pre- and post-treatment means presented in Chapter 5 come from the results included in 
Exhibits 5A.6 and 5A.7. 

• Exhibits 5A.6 and 5A.7: Model with no covariates, only time and treatment variables. 

• Exhibits 5A.8 and 5A.9: Model with covariates used in propensity score model. 

Exhibit 5A.6: Comparative Interrupted Time-Series Estimate of Impact on NSLP Participation 
Rate 



Estimate 

Std Error 

Sig. 

Intercept (ft) 

0.68 

(0.006) 

*** 

Time (ft) 

0.00 

(0.002) 


Treatment LEAs (ft) 

0.04 

(0.009) 

*** 

Post-intervention time period (ft) 

0.00 

(0.004) 


Treatment effect (ft) 

0.04 

(0.004) 

*** 


Dependent variable is average participation rate in the national school lunch program 
Treatment Effect is the interaction of the post-intervention time period and treatment 
***=p<.01, **=p<.05, *= p<.10 


Exhibit 5A.7: Comparative Interrupted Time-Series Estimate of Impact on SBP Participation 
Rate 



Estimate 

Std Error 

Sig. 

Intercept (ft) 

0.37 

(0.011) 

*** 

Time (ft) 

0.00 

(0.004) 


Treatment LEAs (ft) 

0.08 

(0.016) 

*** 

Post-intervention time period (ft) 

0.02 

(0.007) 

** 

Treatment effect (ft) 

0.04 

(0.008) 

*** 


Dependent variable is average participation rate in the School Breakfast Program 
Treatment Effect is the interaction of the post-intervention time period and treatment 
***=p<.01, **=p<.05, *= p<.10 
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Exhibit 5A.8: Comparative Interrupted Time-Series Estimate of Impact on NSLP Participation 
Rate (Model with Covariates) 



Estimate 

Std Error 

Sig. 

Intercept (ft) 

0.41 

(0.027) 

*** 

Time (ft) 

0.00 

(0.002) 


Treatment LEAs (ft) 

0.03 

(0.007) 

*** 

Post-intervention time period (ft) 

0.00 

(0.004) 


Treatment effect (ft) 

0.04 

(0.004) 

*** 

Distribution of ISP a 

ISP indicator: 30%-40% 

0.05 

(0.015) 

*** 

ISP indicator: 40%-50% 

0.03 

(0.016) 

** 

ISP indicator: 50%-60% 

0.08 

(0.017) 

*** 

ISP indicator: 60%-70% 

0.11 

(0.019) 

*** 

ISP indicator > 70% 

0.18 

(0.02) 

*** 

Distributions of Free/Reduced Meals 0 

<50% F/L, <10% R/L 

-0.04 

(0.017) 

** 

>50% F/L, <10% R/L 

0.01 

(0.01) 


<50% F/L, >10% R/L 

0.05 

(0.02) 

*** 

FRP-ISP Difference 

0.30 

(0.042) 

*** 

Percentage of ELL Students 

0.14 

(0.035) 

*** 

Percentage of Students in K-5 grade 0 

0.23 

(0.018) 

*** 

Percentage of Students in 6-8 grade 0 

0.26 

(0.027) 

*** 

LEA in an Urban Area 

-0.02 

(0.01) 

* 

Log of Number of Schools 

-0.02 

(0.005) 

*** 

Avg enrollment per school < 350 

0.01 

(0.007) 

* 

Percentage of African American Students 

-0.02 

(0.014) 

* 

Percentage of Hispanic Students 

-0.09 

(0.029) 

*** 

Percentage of Title 1 Schools 

0.00 

(0.016) 


Michigan Charter Indicator 

-0.06 

(0.015) 

*** 

New York Charter Indicator 

-0.02 

(0.027) 


Ohio Charter Indicator 

-0.05 

(0.016) 

*** 


Dependent variable is average daily participation rate in the National School Lunch Program 
a The reference category is ISP below 20 percent. 

b The reference category is: Greater than 50 percent of reimbursed meals are free; greater than 10 percent of 
reimbursed meals are reduced price. 

c The reference category is the percentage of students in grades 9-12. 

Treatment Effect is the interaction of the post-intervention time period and treatment 
***=p<.01, **=p<05, *= p<.10 
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Exhibit 5A.9: Comparative Interrupted Time-Series Estimate of Impact on SBP Participation 
Rate (Model with Covariates) 



Estimate 

Std Error 

Sig. 

Intercept (ft) 

0.10 

(0.05) 

** 

Time (ft) 

0.00 

(0.004) 


Treatment LEAs (ft) 

0.06 

(0.013) 

*** 

Post-intervention time period (ft) 

0.02 

(0.007) 

** 

Treatment effect (ft) 

0.04 

(0.008) 

■kick 

Distribution of ISP a 

ISP indicator: 30%-40% 

0.06 

(0.028) 

kk 

ISP indicator: 40%-50% 

0.12 

(0.03) 

kkk 

ISP indicator: 50%-60% 

0.18 

(0.032) 

kkk 

ISP indicator: 60%-70% 

0.24 

(0.035) 

kkk 

ISP indicator > 70% 

0.28 

(0.037) 

kkk 

Distributions of Free/Reduced Meals 0 

<50% F/L, <10% R/L 

-0.07 

(0.03) 

kk 

>50% F/L, <10% R/L 

0.03 

(0.018) 


<50% F/L, >10% R/L 

0.03 

(0.037) 


FRP-ISP Difference 

0.12 

(0.077) 


Percentage of ELL Students 

0.31 

(0.063) 

kkk 

Percentage of Students in K-5 grade 0 

0.32 

(0.032) 

kkk 

Percentage of Students in 6-8 grade 0 

0.29 

(0.049) 

kkk 

LEA in an Urban Area 

-0.05 

(0.018) 

kkk 

Log of Number of Schools 

-0.03 

(0.01) 

kkk 

Avg enrollment per school < 350 

0.07 

(0.013) 

kkk 

Percentage of African American Students 

0.06 

(0.026) 

kk 

Percentage of Hispanic Students 

-0.26 

(0.053) 

kkk 

Percentage of Title 1 Schools 

-0.09 

(0.028) 

kkk 

Michigan Charter Indicator 

-0.20 

(0.027) 

kkk 

New York Charter Indicator 

-0.01 

(0.049) 


Ohio Charter Indicator 

-0.12 

(0.029) 

*** 


Dependent variable is average daily participation rate in the School Breakfast Program 
a The reference category is ISP below 20 percent. 

b The reference category is: Greater than 50 percent of reimbursed meals are free; greater than 10 percent of 
reimbursed meals are reduced price. 

c The reference category is the percentage of students in grades 9-12. 

Treatment Effect is the interaction of the post-intervention time period and treatment 
***=p<.01, **=p<.05, *= p<.10 
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5A.2 Analysis of Federal Revenue 

5A.2.1 Baseline Balance Testing for Revenue 

NSLP revenue, as well as the total Federal revenue sample, is identical to the matched administrative 
sample, for which Chapter 2 reported the results of balance testing. However, the matched sample for 
SBP differs due to some missing information. Exhibit 5 A. 10 shows the results of baseline balance 
testing for the SBP revenue sample. 


Exhibit 5A.10: SBP Revenue Sample Characteristics, Weighted to Reflect Many to One 
Matching, LEA Level Measures 



Hypothesis testing performed using t-tests for all outcomes except for grade distribution. 
Grade distribution outcomes jointly tested using MANOVA. 

***=p<.01, **=p<.05, *= p<.10 


5A.2.2 Selection of Baseline Projection Model for Revenue Impact 
Federal Reimbursement Rates for NSLP and SBP 

The first step in the specification of the analysis was the selection of a baseline projection model. The 
evaluation team began by plotting the year-to-year pattern of the mean of Federal reimbursement rate 
for each State separately and for all States combined. These plots were constructed using the 
unadjusted ADP for each LEA aggregated over States. Exhibits 5A. 11 and 5A. 12 present these graphs 
for all States combined. Exhibit 5A.11 depicts the trend in Federal reimbursement rate for the 
National School Lunch Program (NSLP), and Exhibit 5A.12 depicts the trend in Federal 
reimbursement rate for the School Breakfast Program (SBP). Points in the graphs represent the mean 
Federal reimbursement rate for each year measured. As the figures suggest, the Federal 
reimbursement values varied from SY 2009-10 and 2010-11. The rate also exhibits a clear and 
systematic upward trend. 
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Baseline graphs for individual States were inspected for systematic trends that might be masked by 
the combined findings in Exhibits 5A. 11 and 5A.12. Individual States had considerably more year-to- 
year fluctuations in their mean Federal reimbursement rate than was suggested by the combined 
findings. (Exhibits are not shown but are available on request.) However, they also exhibited a clear 
and systematic upward trend in the pre-intervention years. On balance, then, visual inspection of the 
baseline Federal reimbursement rates for the States, both combined and individually, indicated that a 
baseline trend model was appropriate for estimating the impacts of the CEP. Corresponding evidence 
from a statistical analysis of baseline data supports the same conclusion. This evidence was obtained 
by fitting a regression adjusted linear trend model to the pre-intervention Federal reimbursement rates 
for LEAs. Results of that analysis are available on request. 

Exhibit 5A.11: Average Federal Reimbursement Rate for NSLP for Treatment and Comparison 
LEAs over Time 


Avg. Federal Reimbursement per Meal for NSLP for Treatment and Comparison LEAs over time 



♦- Comparisons -•- Treatments 
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Exhibit 5A.12: Average Federal Reimbursement Rate for SBP for Treatment and Comparison 
LEAs over Time 


Avg. Federal Reimbursement per Meal for SBP for Treatment and Comparison LEAs overtime 



*- Comparisons -•- Treatments 


Total Federal Revenue per Student 

Similarly, for total Federal revenue per student enrolled, the evaluation team began by plotting the 
year-to-year pattern for each State separately and for all States combined. Exhibit 5A.13 presents this 
graph for all States combined depicting the trend in total Federal revenue per student enrolled. The 
graph shows a clear and systematic upward trend from SY 2009-10 and 2010-11. Baseline graphs for 
individual States were also inspected and showed considerable more fluctuations than the combined 
findings but still exhibited a clear and systematic upward trend in the pre-intervention years. 
Additionally, evidence from a statistical analysis of baseline data supports the same conclusion 
(Results of that analysis are available on request). 
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Exhibit 5A.13: Average Total Federal Revenue per Student for Treatment and Comparison 
LEAs over Time 


Average Total Federal Revenue for Treatment and Comparison LEAs overtime 
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5A.2.3 Visual Depiction of Federal Revenue Results 
Federal Reimbursement Rates for NSLP and SBP 

As a visual guide for interpreting the CITS impact estimates of CEP on Federal reimbursement rate 
for the NSLP and the SBP, Exhibits 5A.14 and 5A.15 display the regression-adjusted Federal 
reimbursement rate over time for both the treatment and comparison LEAs. In this example, there are 
two pre-treatment years (2009 and 2010) and two post-treatment years (2011 and 2012). The solid 
lines represent the observed fitted regression-adjusted Federal reimbursement rate (blue for treatment 
LEAs and red for comparison LEAs). 4 The dashed line indicates the estimate of the Federal 
reimbursement rate in the absence of the intervention (baseline linear projection—blue for the 
treatment LEAs and red for the comparison LEAs). 5 The observed Federal reimbursement rate for the 
treatment LEAs (represented by the solid differs from the projected reimbursement rate (the 
counterfactual represented by the dashed blue and red lines) in the post-implementation years, and it 
is these differences that allow us to net out the effects of other contextual or maturation processes. 

The difference for the treatment group minus the difference for the comparison group equals the 
estimated impact. 

For the NSLP, the average difference for the treatment group for the Federal reimbursement rate was 
an increase in 10 cents, while the average difference for the comparison group was a decrease of 3 
cents, resulting in a difference-in-difference estimate of 13 cents. For the SBP, the average difference 
for the treatment group for the Federal reimbursement rate was an increase in 1 cent, and the average 
difference for the comparison group was a decrease in 2 cents, and the estimated impact was 3 cents. 


4 For the purposes of the exhibits, the annual means are for Year 1 States (LEAs that adopted the intervention 
in SY 2011-12 and the comparison group in those States). The pre- and post-intervention time periods for 
LEAs that implemented the CEP in 2012 are different, and so the regression-adjusted means may be 
different, but the estimated impact is the same because it derives from the same parameter in the model. 

In the model notation shown in Chapter 5, the intercept for the comparison LEAs is the parameter estimate 
for (/?(,), and the slope is(/? 2 ) • The intercept for treatment LEAs is the estimate (/? 0 + /T) and the slope is 
the same by design (/? 2 )- 
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Exhibit 5A.14: Impact of CEP on Federal Reimbursement Rate for NSLP 


Impact on Fed. Reimbursement per Meal for NSLP 



Time 

Treatment —•- Projected Treatment 

Comparison —•- Projected Comparison 


Exhibit 5A.15: Impact of CEP on Federal Reimbursement Rate for SBP 


Impact on Fed. Reimbursement per Meal for SBP 



Treatment —•- Projected Treatment 

Comparison —•- Projected Comparison 
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Total Federal Revenue per Student 

As a visual guide for interpreting the CITS impact estimates of CEP on total Federal revenue per 
student, Exhibit 5A. 16 displays the regression-adjusted total Federal revenue per student over time for 
both the treatment and comparison LEAs. In this example, there are two pre-treatment years (2009 
and 2010) and two post-treatment years (2011 and 2012). The solid lines represent the observed fitted 
regression-adjusted total Federal revenue per student (blue for treatment LEAs and red for 
comparison LEAs). 6 The dashed line indicates the estimate of the total Federal revenue per student in 
the absence of the intervention (baseline linear projection—blue for the treatment LEAs and red for 
the comparison LEAs). 7 The observed total Federal revenue per student for the treatment LEAs 
(represented by the solid blue) differs from the projected total Federal revenue per student (the 
counterfactual represented by the dashed blue and red lines) in the post-implementation years, and it 
is these differences that allow us to net out the effects of other contextual or maturation processes. 

The difference for the treatment group minus the difference for the comparison group equals the 
estimated impact. The average difference for the treatment group for the total Federal revenue per 
student was an increase in $4.10, and the average difference for the comparison group was a decrease 
in $1.20, and the estimated impact was $5.30. 


6 For the purposes of the exhibits, the annual means are for Year 1 States (LEAs that adopted the intervention 
in SY 2011-12 and the comparison group in those States). The pre- and post-intervention time periods for 
LEAs that implemented the CEP in 2012 are different, and so the regression-adjusted means may be 
different, but the estimated impact is the same because it derives from the same parameter in the model. 

7 In the model notation shown in Chapter 5, the intercept for the comparison LEAs is the parameter estimate 
for (/? 0 ), and the slope is(/? 2 ) • The intercept for treatment LEAs is the estimate (/? 0 + f } ± ) and the slope is 
the same by design (/? 2 )- 
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Exhibit 5A.16: Impact of CEP on Total Federal Revenue per Student 



Treatment —•- Projected Treatment 

Comparison —•- Projected Comparison 


5A.2.4 Regression Results for Federal Revenue 
Federal Reimbursement Rates for NSLP and SBP 

While Chapter 5 specifically reports the estimated impacts, this Appendix exhibits presents all of the 
parameter estimates from the models used to estimate impacts on Federal reimbursement rate for 
NSLP and SBP. The estimated pre- and post-treatment means presented in Chapter 5 come from the 
results included in Exhibits 5A. 17 and 5A. 18. 

• Exhibits 5A.17 and 5A. 18: Model with no covariates, only time and treatment variables. 

• Exhibits 5 A. 19 and 5A.20: Model with covariates used in propensity score model. 


Exhibit 5A.17: Comparative Interrupted Time-Series Estimate of Impact on Federal 
Reimbursement Rate for NSLP 



Estimate 

Std Error 


Intercept (ft) 

2.26 

(0.02) 

*** 

Time (ft) 

0.11 

(0.004) 

*** 

Treatment LEAs (ft) 

0.13 

(0.032) 

*** 

Post-intervention time period (ft) 

-0.05 

(0.009) 

*** 

Treatment effect (ft) 

0.13 

(0.01) 

*** 


Dependent variable is average Federal reimbursement rate for the national school lunch program 
Treatment Effect is the interaction of the post-intervention time period and treatment 
***=p<.01, **=p<05, *= p<.10 
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Exhibit 5A.18: Comparative Interrupted Time-Series Estimate of Impact on Federal 
Reimbursement Rate for SBP 



Estimate 

Std Error 


Intercept (ft) 

1.62 

(0.009) 

*** 

Time (ft) 

0.05 

(0.004) 

*** 

Treatment LEAs (ft) 

0.03 

(0.013) 

** 

Post-intervention time period (ft) 

-0.02 

(0.008) 

*** 

Treatment effect (ft) 

0.03 

(0.009) 

*** 


Dependent variable is average Federal reimbursement rate for the School Breakfast Program 
Treatment Effect is the interaction of the post-intervention time period and treatment 
***=p<.01, **=p<.05, *= p<.10 
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Exhibit 5A.19: Comparative Interrupted Time-Series Estimate of Impact on Federal 
Reimbursement Rate for NSLP (Model with Covariates) 



Estimate 

Std Error 

Sig. 

Intercept (ft) 

1.65 

(0.052) 

*** 

Time (ft) 

0.11 

(0.004) 

*** 

Treatment LEAs (ft) 

0.05 

(0.013) 

*** 

Post-intervention time period (ft) 

-0.05 

(0.009) 

*** 

Treatment effect (ft) 

0.13 

(0.01) 

*** 

Distribution of ISP a 

ISP indicator: 30%-40% 

0.13 

(0.028) 

*** 

ISP indicator: 40%-50% 

0.34 

(0.03) 

*** 

ISP indicator: 50%-60% 

0.46 

(0.032) 

*** 

ISP indicator: 60%-70% 

0.55 

(0.035) 

*** 

ISP indicator > 70% 

0.63 

(0.038) 

*** 

Distributions of Free/Reduced Meals 0 

<50% F/L, <10% R/L 

-0.41 

(0.032) 

*** 

>50% F/L, <10% R/L 

0.02 

(0.019) 


<50% F/L, >10% R/L 

-0.24 

(0.039) 

*** 

FRP-ISP Difference 

0.58 

(0.081) 

*** 

Percentage of ELL Students 

0.11 

(0.065) 

* 

Percentage of Students in K-5 grade 0 

0.04 

(0.033) 


Percentage of Students in 6-8 grade 0 

-0.03 

(0.051) 


LEA in an Urban Area 

0.00 

(0.019) 


Log of Number of Schools 

0.04 

(0.01) 

*** 

Avg enrollment per school < 350 

0.03 

(0.014) 

** 

Percentage of African American Students 

0.21 

(0.027) 

*** 

Percentage of Hispanic Students 

0.16 

(0.055) 

*** 

Percentage of Title 1 Schools 

-0.02 

(0.03) 


Michigan Charter Indicator 

0.10 

(0.028) 

*** 

New York Charter Indicator 

0.12 

(0.051) 

** 

Ohio Charter Indicator 

0.29 

(0.03) 

*** 


Dependent variable is average Federal reimbursement rate for the National School Lunch Program 
a The reference category is ISP below 20 percent. 

b The reference category is: Greater than 50 percent of reimbursed meals are free; greater than 10 percent of 
reimbursed meals are reduced price. 

c The reference category is the percentage of students in grades 9-12. 

Treatment Effect is the interaction of the post-intervention time period and treatment 
***=p<.01, **=p<05, *= p<.10 
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Exhibit 5A.20: Comparative Interrupted Time-Series Estimate of Impact on Federal 
Reimbursement Rate for SBP (Model with Covariates) 



Estimate 

Std Error 

Sig. 

Intercept ((3 0 ) 

1.35 

(0.039) 

*** 

Time (M 

0.05 

(0.004) 

*** 

Treatment LEAs ((S 2 ) 

0.02 

(0.01) 


Post-intervention time period (p 3 ) 

-0.02 

(0.008) 

*** 

Treatment effect (f> 4 ) 

0.03 

(0.009) 

*** 

Distribution of ISP a 

ISP indicator: 30%-40% 

0.04 

(0.021) 

* 

ISP indicator: 40%-50% 

0.03 

(0.023) 


ISP indicator: 50%-60% 

0.08 

(0.025) 

*** 

ISP indicator: 60%-70% 

0.11 

(0.026) 

*** 

ISP indicator > 70% 

0.20 

(0.029) 

*** 

Distributions of Free/Reduced Meals 0 

<50% F/L, <10% R/L 

-0.13 

(0.024) 

*** 

>50% F/L, <10% R/L 

0.04 

(0.014) 

*** 

<50% F/L, >10% R/L 

-0.18 

(0.029) 

*** 

FRP-ISP Difference 

0.48 

(0.06) 

*** 

Percentage of ELL Students 

0.03 

(0.048) 


Percentage of Students in K-5 grade 0 

0.02 

(0.025) 


Percentage of Students in 6-8 grade 0 

-0.06 

(0.039) 


LEA in an Urban Area 

0.00 

(0.014) 


Log of Number of Schools 

0.01 

(0.007) 


Avg enrollment per school < 350 

-0.03 

(0.01) 

*** 

Percentage of African American Students 

0.01 

(0.019) 


Percentage of Hispanic Students 

-0.02 

(0.04) 


Percentage of Title 1 Schools 

0.08 

(0.022) 

*** 

Michigan Charter Indicator 

-0.02 

(0.02) 


New York Charter Indicator 

0.01 

(0.038) 


Ohio Charter Indicator 

0.09 

(0.022) 

*** 


Dependent variable is average Federal reimbursement rate for the School Breakfast Program 
a The reference category is ISP below 20 percent. 

b The reference category is: Greater than 50 percent of reimbursed meals are free; greater than 10 percent of 
reimbursed meals are reduced price. 

c The reference category is the percentage of students in grades 9-12. 

Treatment Effect is the interaction of the post-intervention time period and treatment 
***=p<.01, **=p<.05, *= p<.10 

Total Federal Revenue per Student 

While Chapter 5 specifically reports the estimated impacts, this Appendix exhibits presents all of the 
parameter estimates from the models used to estimate impacts on total Federal revenue per student. 
The estimated pre- and post-treatment means presented in Chapter 5 come from the results included 
in Exhibits 5A.21. 

• Exhibits 5A.21: Model with no covariates, only time and treatment variables. 

• Exhibits 5A.22: Model with covariates used in propensity score model. 
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Exhibit 5A.21: Comparative Interrupted Time-Series Estimate of Impact on Total Federal 
Revenue per Student 



Estimate 

Std Error 

Sig. 

Intercept (ft) 

38.14 

(0.685) 

*** 

Time (ft) 

2.67 

(0.192) 

*** 

Treatment LEAs (ft) 

5.03 

(1.036) 

*** 

Post-intervention time period (ft) 

-1.16 

(0.425) 

*** 

Treatment effect (ft) 

5.33 

(0.46) 

*** 


Dependent variable is average total revenue per student enrolled 

Treatment Effect is the interaction of the post-intervention time period and treatment 

***=p<.01, **=p< 05, *= p<.10 


Exhibit 5A.22: Comparative Interrupted Time-Series Estimate of Impact on Total Federal 
Revenue per Student (Model with Covariates) 



Estimate 

Std Error 

Sig. 

Intercept (ft) 

7.83 

(2.441) 

*** 

Time (ft) 

2.74 

(0.19) 

*** 

Treatment LEAs (ft) 

2.78 

(0.61) 

*** 

Post-intervention time period (ft) 

-1.20 

(0.423) 

*** 

Treatment effect (ft) 

5.18 

(0.453) 

*** 

Distribution of ISP a 

ISP indicator: 30%-40% 

4.17 

(1.3) 

*** 

ISP indicator: 40%-50% 

8.86 

(1.407) 

*** 

ISP indicator: 50%-60% 

13.87 

(1.501) 

*** 

ISP indicator: 60%-70% 

16.68 

(1.633) 

*** 

ISP indicator > 70% 

24.18 

(1.747) 

*** 

Distributions of Free/Reduced Meals 0 

<50% F/L, <10% R/L 

-6.65 

(1.49) 

*** 

>50% F/L, <10% R/L 

0.59 

(0.88) 


<50% F/L, >10% R/L 

-2.12 

(1.793) 


FRP-ISP Difference 

21.12 

(3.765) 

*** 

Percentage of ELL Students 

16.82 

(3.047) 

*** 

Percentage of Students in K-5 grade 0 

19.16 

(1.546) 

*** 

Percentage of Students in 6-8 grade 0 

18.55 

(2.379) 

*** 

LEA in an Urban Area 

-1.39 

(0.871) 


Log of Number of Schools 

-0.09 

(0.458) 


Avg enrollment per school < 350 

2.79 

(0.644) 

*** 

Percentage of African American Students 

4.91 

(1.248) 

*** 

Percentage of Hispanic Students 

-7.23 

(2.548) 

*** 

Percentage of Title 1 Schools 

0.42 

(1.41) 


Michigan Charter Indicator 

-8.62 

(1.29) 

*** 

New York Charter Indicator 

6.04 

(2.358) 

** 

Ohio Charter Indicator 

1.49 

(1.395) 



Dependent variable is average total revenue per student enrolled 
a The reference category is ISP below 20 percent. 

b The reference category is: Greater than 50 percent of reimbursed meals are free; greater than 10 percent of 
reimbursed meals are reduced price. 

c The reference category is the percentage of students in grades 9-12. 

Treatment Effect is the interaction of the post-intervention time period and treatment 
***=p<.01, **=p<05, *= p<.10 
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5A.3 Analysis of Non-Federal Revenues 

5A.3.1 Non-Federal Revenue Outcomes, Data Sources and the Analytic Sample 
State Revenues 

State revenues were obtained from the PEAR survey for States other than Illinois. PEAR survey 
respondent LEAs that indicated they received revenue from State funds and were able to report the 
amount received were asked to provide the amount of revenue received from State sources. In States 
other than Illinois, only individuals that provide a dollar figure for State revenue received were 
included in the analysis. 

State revenues from Illinois are obtained from State data. To avoid distorting the Component 1 PEAR 
Survey Sample, only State Revenues received by PEAR survey respondent LEAs are included in the 
analysis. 

Local Revenues 

Local revenues were obtained from the PEAR survey for all States. As described for the State 
revenues, PEAR survey respondents were first asked if such revenue was received by the LEA. If 
revenues were received, the FSD was asked whether he or she could report the revenue separately 
from other revenues. The survey collected data on the specific amount of local funds received only if 
the revenue could be reported separately. Because such a large proportion of LEAs report that the 
foodservice program did not receive funds from local sources (77 percent of LEAs in 2010-11), the 
evaluation team decided to focus on the question, “Did the CEP increase the probability that the 
foodservice program receive any funding from local sources?” This focus allowed the sample to 
include all LEAs that responded to the initial question regarding the generation of revenue, the largest 
possible sample, and it captures the key variation observed in the data. 

Non-Reimbursable Foods Revenues 

Revenues from payments for non-reimbursable foods, including competitive foods, a la carte, vending 
machines and adult foods were obtained from the PEAR survey for all States other than Kentucky. 

The structure of these data are similar to local revenue data. PEAR survey respondents were first 
asked if Non-reimbursable Foods Revenue was received by the LEA. If revenues were received, the 
FSD was asked whether he or she could report the revenue separately from other revenues. The 
survey collected data on the specific amount of revenue generated from Non-Reimbursable Foods 
only if the revenue could be reported separately. Because such a large proportion of LEAs report that 
no such revenue was generated (66 percent of LEAs in 2010-11), the evaluation team decided to 
focus on the question, “Did the CEP increase the probability that the foodservice program generated 
revenue from Non-Reimbursable Foods?”, as was decided for local revenues. 

The analysis of Non-Reimbursable Foods Revenues excludes all LEAs from Kentucky. Kentucky 
provided State Administrative Data on these revenues for the 2009-10, 2010-11 and 2011-2012 
school years. However, these data were not provided for the final year 2012-13. Because most of the 
Treatment LEAs in the PEAR survey sample did not take up the CEP until the 2012-13 school year, 
the Administrative Data was missing data that reflected the revenues generated under the CEP for 20 
out of 24 Treatment LEAs. 
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Student Revenues 

Revenues from student payments for reimbursable meals were obtained from the PEAR survey for all 
States other than Kentucky. As described in Chapter 5, revenues from Student Payments for 
Reimbursable Meals are measured by the fall total revenue received from student payments per 
student for LEAs that report such revenue and was defined to be zero for LEAs that report that no 
revenue was generated from student payments. This outcome measure therefore includes all LEAs 
that reported that they did not generate revenue from student payments and all LEAs that reported the 
fall revenue from student payments. The outcome measure excludes all LEAs that indicated that 
revenue was generated from student payments and that did not report fall student revenue, either 
because the LEA was unable to report student revenue separately or because the fall student revenue 
data were not available. 

Exhibit 5A.23 below describes the number of LEAs that reported generating no student revenue, the 
number that indicated generating student revenue but were unable to report fall student revenue, and 
the number that indicated that generating student revenue and reported fall student revenue. The 
figures include all LEAs that provided a yes or no response to the data item on generating student 
revenue. In the 2010-11 school year, 62 Comparison and 86 Treatment LEAs provided such a 
response. Out of these LEAs, 4 Comparison and 15 Treatment LEAs (6 percent and 17 percent, 
respectively) report generating no revenue from student payments for reimbursable meals. Elowever, 
the outcome variable is undefined for LEAs that generated student revenue and are unable to report 
fall student revenue. Lienee, among LEAs with defined Student Revenue outcomes, the proportion of 
LEAs that reported generating no student revenue approximately doubles, accounting for 13 percent 
of Comparison and 33 percent of Treatment LEAs. See Chapter 5 for a discussion of how this 
overrepresentation of LEAs that generated no student revenue biases results. 


Exhibit 5A.23: Number of LEAs that Generated and Reported Student Revenue, by Treatment 
Status 



Comparison 

Treatment 

2009-10 School Year 

No Student Revenue Generated 

7 

11 

Generated Student Revenue & Unable to Report Fall Student Revenue 

32 

36 

Reported Fall Student Revenue 

20 

33 

2010-11 School Year 

No Student Revenue Generated 

4 

15 

Generated Student Revenue & Unable to Report Fall Student Revenue 

31 

40 

Reported Fall Student Revenue 

27 

31 

2011-12 School Year 

No Student Revenue Generated 

5 

21 

Generated Student Revenue & Unable to Report Fall Student Revenue 

32 

35 

Reported Fall Student Revenue 

25 

31 

2012-13 School Year 

No Student Revenue Generated 

7 

43 

Generated Student Revenue & Unable to Report Fall Student Revenue 

19 

16 

Reported Fall Student Revenue 

34 

28 


Source: PEAR Survey 


As for analysis of Non-Reimbursable Foods Revenues, the State Revenue analysis excludes all LEAs 
from Kentucky. Kentucky provided State Administrative Data on these revenues for the 2009-10, 
2010-11 and 2011-12 school years. However, these data were not provided for the final year 2012- 
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13. Because most of the Treatment LEAs in the PEAR survey sample did not take up the CEP until 
the 2012-13 school year, the Administrative Data was missing data that reflected the revenues 
generated under the CEP for 20 out of 24 Treatment LEAs. 
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Exhibit 5A.24: Baseline Balance Tests on Analytic Sample for State and Local Revenue Analysis 




State Revenue Analysis 



Local Revenue Analysis 



Comparison 

Treatment 

Comparison 

Treatment 


Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Number of observations 

45 


54 


64 


97 


Enrollment 

3,418 

(6,058) 

3,066 

(5,728) 

3,400 

(5,575) 

3,374 

(5,565) 

Enrollment 1-499 (%) 

22.2 


29.6 


25.0 


27.8 


Enrollment 500-2,499 (%) 

44.4 


40.7 


39.1 


32.0 


Enrollment 2,500-4,999 (%) 

17.8 


14.8 


18.8 


21.6 


Enrollment 5,000+ (%) 

15.6 


14.8 


17.2 


18.6 


Percentage distribution of students in grades K-12 

Percent in grades K-5 

48.3 

(14.8) 

50.9 

(15.0) 

49.1 

(17.9) 

47.9 

(17.1) 

Percent in grades 6-8 

24.3 

(6.2) 

23.4 

(7.2) 

22.8 

(7.1) 

21.2 

(7.8) 

Percent in grades 9-12 

22.5 

(20.3) 

21.4 

(21.1) 

24.6 

(18.9) 

27.4 

(21.3) 

Number of schools 

6.9 

(9.1) 

7.1 

(11.1) 

6.7 

(8.8) 

7.5 

(10.3) 

1 school (%) 

18.2 


28.3 


25.4 


25.3 


2-5 schools (%) 

45.5 


35.8 


39.7 


29.5 


6-14 schools (%) 

25.0 


22.6 


25.4 


30.5 


15+ schools (%) 

11.4 


11.3 


9.5 


12.6 


Identified Student Percentage (ISP) 

43.7 

(16.1)*** 

52.9 

(15.9)*** 

45.9 

(14.8) 

49.3 

(16.1) 

Percent students free/reduced lunch 

68.7 

(15.4) 

72.4 

(15.3) 

68.3 

(16.3) 

69.4 

(15.8) 

Percent Title 1 schools 

84.2 

(19.8)** 

92.0 

(13.8)** 

81.2 

(22.9) 

84.7 

(20.1) 

Urban LEA (%) 

20.0 


29.6 


26.6 


25.8 


All charter schools (%) 

6.7 

* 

18.5 

* 

20.3 


21.6 


Percent English Language Learners 

3.6 

(5.7) 

2.5 

(4.8) 

2.6 

(5.9) 

2.1 

(5.2) 

Percent students Black 

21.8 

(26.8) 

32.7 

(38.) 

26.4 

(33.4) 

26.1 

(35.8) 

Percent students Hispanic/Latino 

11.2 

(14.9)* 

6.4 

(9.4)* 

6.5 

(10.4) 

6.0 

(11.2) 

Average Daily Participation (%) Lunch 

71.3 

(10.6) 

71.3 

(13.5) 

70.0 

(11.7) 

69.0 

(13.5) 

Average Daily Participation (%) Breakfast 3 

46.4 

(30.0) 

56.1 

(35.0) 

40.3 

(27.6) 

43.9 

(26.7) 


Hypothesis testing performed using t-tests for all outcomes except for grade distribution. Grade distribution outcomes jointly tested using MANOVA. 
*** Difference is significant at the 1% level; ** Difference is significant at the 5% level; * Difference is significant at the 10% level 
3 Observed for all LEAs in the State Revenue Analysis. Observed for 155 LEAs in the Local Revenue Analysis (59 Comparison and 96 Treatment). 
Source: Common Core of School Data, State-Provided Data on SBP and NSLP Participation, PEAR Survey 
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Exhibit 5A.25: Baseline Balance Tests on Analytic Sample for Analysis of Revenue from Payments for Non-Reimbursable Foods and 
Student Payments for Reimbursable Foods 



Non-Reimbursable Foods Revenue Analysis 

Student Payments for Reimbursable Foods 
Revenue Analysis 

Comparison 

Treatment 

Comparison 

Treatment 


Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Number of observations 

61 


86 


29 


53 


Enrollment 

2,724 

(3,557) 

2,869 

(4,800) 

2,764 

(3,436) 

3,120 

(5,731) 

Enrollment 1-499 (%) 

32.8 


32.6 


31.0 


39.6 


Enrollment 500-2,499 (%) 

37.7 


31.4 


41.4 


26.4 


Enrollment 2,500-4,999 (%) 

14.8 


20.9 


13.8 


17.0 


Enrollment 5,000+ (%) 

14.8 


15.1 


13.8 


17.0 


Percentage distribution of students in grades K-12 

Percent in grades K-5 

51.2 

(19.9) 

48.6 

(18.4) 

48.9 

(18.6) 

48.3 

(19.5) 

Percent in grades 6-8 

23.0 

(7.7) 

21.5 

(8.6) 

23.9 

(7.1) 

22.0 

(9.6) 

Percent in grades 9-12 

22.6 

(21.2) 

26.1 

(23.3) 

23.6 

(22.9) 

26.1 

(26.0) 

Number of schools 

5.7 

(6.6) 

6.7 

(8.9) 

5.9 

(7.0) 

7.2 

(11.2) 

1 school (%) 

31.7 


30.1 


28.6 


35.3 


2-5 schools (%) 

36.7 


27.7 


35.7 


27.5 


6-14 schools (%) 

23.3 


26.5 


21.4 


17.6 


15+ schools (%) 

8.3 


12.0 


14.3 


15.7 


Identified Student Percentage (ISP) 

47.7 

(16.6) 

51.8 

(16.4) 

45.7 

(18.1)“ 

54.4 

(15.4)“ 

Percent students free/reduced lunch 

70.5 

(17.4) 

70.6 

(16.9) 

67.0 

(16.8) 

73.0 

(16.1) 

Percent Title 1 schools 

84.3 

(23.7) 

85.0 

(21.3) 

78.4 

(25.3)* 

87.4 

(20.8)* 

Urban LEA (%) 

31.1 


30.2 


20.7 


35.8 


All charter schools (%) 

24.6 


26.7 


20.7 


32.1 


Percent English Language Learners 

3.2 

(6.7) 

2.4 

(5.7) 

2.1 

(4.2) 

2.1 

(4.7) 

Percent students Black 

31.2 

(34.3) 

31.5 

(37.9) 

26.2 

(32.7) 

38.4 

(39.2) 

Percent students Hispanic/Latino 

9.2 

(14.2) 

6.8 

(12.3) 

5.9 

(9.1) 

6.1 

(8.9) 

Average Daily Participation (%) Lunch 

68.6 

(11.8) 

67.3 

(13.7) 

67.1 

(11.4) 

67.7 

(14.2) 

Average Daily Participation (%) Breakfasta 

43.6 

(29.6) 

46.4 

(28.6) 

44.6 

(28.8) 

48.2 

(30.0) 


Hypothesis testing performed using t-tests for all outcomes except for grade distribution. Grade distribution outcomes jointly tested using MANOVA. 
***=p<.01, **=p<05, *= p<.10 


a Observed for 141 LEAs in the Non-Reimbursable Foods Revenue Analysis (56 Comparison and 85 Treatment), and for 79 LEAs in the Student Payments for 
Reimbursable Foods Revenue Analysis (27 Comparison and 52 Treatment). 

Source: Common Core of School Data, State-Provided Data on SBP and NSLP Participation, PEAR Survey 
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Exhibit 5A.26: Detailed Results for Analysis of Non-Federal Revenues 


(D 

(2) 

(3) 

Outcome 

State Revenue 

Local Revenue 

Local Revenue 

Estimates 

Intercept (ft) 

4.34 

0.27*** 

-6.17*** 


(18.60) 

(0.04) 

(0.85) 

Treatment LEAs (/?2 ) 

8.55 

-0.02 

0.55 


(7.28) 

(0.06) 

(0.99) 

Post-intervention time period (ft) 

-1.11 

-0.01 

-0.23 


(2.00) 

(0.02) 

(0.59) 

Treatment effect (ft) 

2.49 

-0.02 

-0.65 


(2.77) 

(0.03) 

(0.78) 

Identified Student Percentage (ISP) 

-14.69 



(23.75) 

Percent Title 1 schools 

9.87 



(21.49) 

All charter schools (%) 

14.63 



(11.05) 

Percent students Hispanic/Latino 

5.95 



(28.18) 

Model 

CITS 

CITS—Linear 
Probability 

CITS—Logistic 
Regression 

LEA-level Random Effect 

Yes 

Yes 

Yes 

Observations 

360 

672 

614 

Number of LEAs 

99 

200 

161 


***=p<.01, **=p<05, *= p<.10 


Exhibit 5A.26 Continued: Detailed Results for Analysis of Non-Federal Revenues 


Outcome 

(4) 

(5) 

(6) 

Non- 

Reimbursable 

Foods 

Non- 

Reimbursable 

Foods 

Student 

Payments 

Estimates 

Intercept (ft) 

0.66*** 

4.61*** 

69.13*** 


(0.06) 

(0.58) 

(15.00) 

Treatment LEAs (ft) 

0.03 

5.70*** 

10.63 


(0.08) 

(1.16) 

(8.26) 

Post-intervention time period (ft) 

-0.03 

-0.69 

-4.58 


(0.02) 

(0.66) 

(6.64) 

Treatment effect (ft) 

-0.03 

-2.08* 

-8.24 


(0.03) 

(1.08) 

(8.39) 

Identified Student Percentage (ISP) 



-54.59“ 

(23.11) 

Percent Title 1 schools 



-28.93* 

(16.99) 

Model 

CITS—Linear 

CITS—Logistic 

CITS 


Probability 

Regression 


LEA-level Random Effect 

Yes 

Yes 

Yes 

Observations 

557 

557 

291 

Number of LEAs 

147 

147 

82 


***=p<.01, **=p<05, *= p<.10 
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Appendix 6A: Analysis of Impacts on Administrative Costs and 
Staffing 


This appendix presents supporting information for the discussion in Chapter 6 about the impacts of 
the CEP on administrative costs and staffing at the LEA and school levels. The appendix begins with 
information on the data and results of balance testing. Next, univariate statistics for outcome variables 
are presented. The appendix concludes with the complete regression results supporting the impact 
estimates. 


6A.1 Data and Balance Testing 

The analysis file for the cost analysis was constructed by merging data from LEA and school staff 
rosters and administrative cost interviews, completed as part of the data collection for Component 2 
of the Impact Study, described in Chapter 2. Three comparison LEAs selected for this data collection 
refused to complete staff rosters or participate in the cost interviews. Therefore, the sample was 
drawn from 100 LEAs (52 treatment and 48 comparison) rather than the 103 LEAs in the full 
Component 2 sample. Exhibit 6A.1 repeats the analysis from Exhibit 2.7 from Chapter 2 for this 
somewhat smaller set of LEAs. The substantive conclusions from Chapter 2 are unaltered: treatment 
and comparison LEAs for the cost analysis are balanced on all LEA-level observed covariates with 
the exception of ISP. Because the ISP was unbalanced between treatment and comparison LEAs, it 
was including as a control variable in the impact estimation model. 


Abt Associates 
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Exhibit 6A.1: LEA Characteristics for Administrative Cost Analysis Sample 



Comparison 

Mean (SD) 

Treatment 

Mean (SD) 

Enrollment 

4,062 

(5869) 

4,060 

(6955) 

Enrollment 1-499 (%) 

16.7 


21.2 


Enrollment 500-2,499 (%) 

43.8 


36.5 


Enrollment 2,500^1,999 (%) 

14.6 


25 


Enrollment 5,000+ (%) 

25 


17.3 


Percentage distribution of students in grades K-12 

Percent in grades K-5 

48.9 

(16.7) 

51 

(13.9) 

Percent in grades 6-8 

20.7 

(6.3) 

23.5 

(7.9) 

Percent in grades 9-12 

26.8 

(17.3) 

21.7 

(13.5) 

Number of schools 

7.8 

(8.9) 

8.9 

(12.8) 

1 school (%) 

14.6 


17.6 


2-5 schools (%) 

37.5 


37.3 


6-14 schools (%) 

37.5 


27.5 


15+ schools (%) 

10.4 


13.7 


ISP 

47.2 

(15.4)** 

55.2 

(16.5)** 

Percent students free/reduced lunch 

69.9 

(15.0) 

74.3 

(14.6) 

Percent Title 1 schools 

85.6 

(21.6) 

87.1 

(15.1) 

Urban LEA (%) 

25 


30.8 


All charter schools (%) 

12.5 


19.2 


Percent English Language Learners 

3.2 

(5.3) 

3.4 

(6.9) 

Percent students Black 

26.9 

(31.8) 

33.5 

(35.9) 

Percent students Hispanic/Latino 

8.1 

(11.8) 

8.9 

(15.0) 

Average Daily Participation (%) Lunch 

67.4 

(11.1) 

69.9 

(12.8) 

Number of LEAs 

48 


52 



T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade distribution. Differences in grade distributions were jointly tested using MANOVA. 

Source: Common Core of School Data, State Administrative Data, Verification Summary Report Data 
***=p<.01, **=p<05, *= p<.10 

Data were collected from 247 schools in the 100 LEAs that agreed to participate in the administrative 
cost data collection. Characteristics of these schools are shown in Exhibit 6A.2. Treatment schools 
had somewhat lower average enrollment then comparison schools (473 versus 588) and more students 
eligible for free and reduced-price meals (70.4 versus 65.9 percent). Treatment and comparison 
schools were otherwise statistically indistinguishable. Because enrollment and the percentage of 
students eligible for free and reduced-price meals were unbalanced between treatment and 
comparison schools, they were included as control variables in the impact estimation model. 
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Exhibit 6A.2: School-level Characteristics for the Administrative Cost Analysis Sample 

Comparison schools Treatment schools 


Mean (SD)Mean (SD) 


Enrollment 

588 

(421)** 

473 

(315)** 

Type of school 

Title 1 (%) 

84.7 

(36.2) 

91.1 

(28.7) 

Charter (%) 

4.8 

(21.6) 

9.8 

(29.8) 

In city (%) 

21.8 

(41.4) 

27.6 

(44.9) 

Student characteristics 

Free/reduced lunch (%) 

65.9 

(16.2)** 

70.4 

(18.6)** 

Black (%) 

21.7 

(28.6) 

27.2 

(33.2) 

Hispanic/Latino (%) 

7.4 

(10.7) 

8.7 

(15.4) 

Grade span a 

K-5 (%) 

34.7 

(47.8) 

35.8 

(48.1) 

6-8 (%) 

29.0 

(45.6) 

25.2 

(43.6) 

9-12(%) 

29.0 

(45.6) 

22.8 

(42.1) 

Other (%) 

7.3 

(26.1) 

16.3 

(37.1) 

Grades Sampled a 

K-5 (%) 

38.7 

(48.9) 

45.5 

(50.0) 

6-8 (%) 

32.3 

(46.9) 

30.9 

(46.4) 

9-12(%) 

29.0 

(45.6) 

23.6 

(42.6) 

Number of schools 

124 


123 



T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade span and grades sampled. Differences in grade span and grades sampled were jointly tested using a chi- 
square test. 

Source: Common Core of School Data 
***=p<.01, **=p<05, *= p<.10 

a The grade span refers to the actual grade coverage of a school. Grade coverage can be irregular, especially 
among charter schools, which may cover K-12, K-8, or other grade spans, all of which are included in the “Other” 
category in the table. The analysis has categorized these irregular grade spans according to three separate 
sampled grades: elementary (K-5), middle (6-8), and high (9-12). The table presents both the original grade span 
and the sampled grade category. 


6A.2 Univariate Statistics for Outcome Variables 


Exhibit 6A.3 shows the means and standard deviation for the outcome variables used in the analysis 
of CEP impacts on administrative labor time and costs by task domain. Exhibit 6A.4 does the same 
by staff type. 
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Exhibit 6A.3: Univariate Statistics for Annual Time and Labor Costs for Administrative Tasks 
Potentially Affected by CEP Participation, by Task Domain 



Comparison schools 

Treatment schools 


Mean (SD) 

Mean (SD) 

Hours per enrolled student 

A. Distributing and processing applications for free 
or reduced-price meals 

0.46 

(0.55) 

0.12 

(0.31)*“ 

B. Direct Certifications 

0.09 

(0.22) 

0.12 

(0.24) 

C. Verifying income of free/reduced price students 

0.12 

(0.31) 

0.01 

(0.06)*“ 

D. Meal payment collections and accounting 

1.92 

(1.63) 

1.38 

(2.07)“ 

E. Counting and claiming reimbursable meals 

0.43 

(0.82) 

1.06 

(1.47)“* 

Total 

3.01 

(2.49) 

2.69 

(2.86) 

Cost per enrolled student 

A. Distributing and processing applications for free 
or reduced-price meals 

13.78 

(14.54) 

3.37 

(8.04)*** 

B. Direct Certifications 

3.45 

(10.14) 

4.32 

(10.86) 

C. Verifying income of free/reduced price students 

3.99 

(9.70) 

0.47 

(1.49)*** 

D. Meal payment collections and accounting 

41.71 

(31.79) 

30.61 

(46.63)“ 

E. Counting and claiming reimbursable meals 

9.45 

(16.09) 

26.28 

(36.89)“* 

Total 

72.38 

(56.04) 

65.06 

(69.41) 


T-tests were used to test for differences between treatment and comparison means for all outcomes 
***=p<.01, *‘=p< 05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.4: Univariate Statistics for Annual Time and Labor Costs for Administrative Tasks 
Potentially Affected by CEP Participation, by Staff Type 



Comparison schools 

Treatment schools 

Mean (SD) 

Mean (SD) 

Hours per enrolled student 

A. LEA staff 

0.54 

(0.80) 

0.54 

(1.02) 

B. School foodservice staff 

1.77 

(1.79) 

1.67 

(2.54) 

C. School non-foodservice staff 

0.56 

(1.90) 

0.41 

(1.04) 

D. School staff, role unknown 

0.14 

(0.61) 

0.07 

(0.35) 

Total 

3.01 

(2.49) 

2.69 

(2.86) 

Cost per enrolled student 

A. LEA staff 

19.03 

(26.80) 

15.22 

(23.54) 

B. School foodservice staff 

35.91 

(33.69) 

31.97 

(52.18) 

C. School non-foodservice staff 

13.83 

(39.77) 

15.34 

(38.27) 

D. School staff, role unknown 

3.61 

(15.71) 

2.53 

(13.89) 

Total 

72.38 

(56.04) 

65.06 

(69.41) 


T-tests were used to test for differences between treatment and comparison means for all outcomes 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


6A.3 Regression Results 


Exhibits 6A.5-24 present the complete set of regression results. The vector of school characteristics 
used as control variables in all models included: 

• the school’s grade level (i.e., whether elementary, middle, or high), with high school as the 
reference category; 
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• the State, with West Virginia as the reference State; and 

• LEA and school-level features where balance testing (detailed above) discerned a difference 
between treatment and comparison LEAs that was statistically significant at the 10 percent level, 
which included school enrollment and FRP participation, as well as the LEA’s ISP. 


Exhibit 6A.5: Administrative Time and Cost Regression Results for Hours per Student, Total 



Coef. 

SE 

Sig. 

Treatment 

-0.69 

0.42 


Enrollment (in 1000s) 

-1.55 

0.55 

*** 

% FRP 

1.71 

1.31 


LEA ISP 

0.20 

2.17 


Middle school grades sampled 

-0.68 

0.30 

** 

High school grades sampled 

-0.08 

0.37 


Illinois 

0.59 

0.87 


Kentucky 

0.99 

0.78 


Michigan 

-0.88 

0.63 


New York 

-0.52 

0.83 


Ohio 

-0.13 

1.16 


Constant 

3.02 

1.01 

*** 

Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.6: Administrative Time and Cost Regression Results for Cost per Student, Total 



Coef. 

SE 

Sig. 

Treatment 

-16.72 

10.10 


Enrollment (in 1000s) 

-33.17 

13.88 

** 

% FRP 

38.69 

30.93 


LEA ISP 

22.32 

55.18 


Middle school grades sampled 

-15.71 

6.08 

** 

High school grades sampled 

-3.97 

8.85 


Illinois 

-3.55 

23.74 


Kentucky 

-13.33 

22.13 


Michigan 

-49.55 

18.76 

*** 

New York 

-40.51 

20.35 

** 

Ohio 

-32.65 

26.61 


Constant 

88.21 

26.06 

*** 

Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 
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Exhibit 6A.7: Administrative Time and Cost Regression Results for Hours per Student, LEA 
Staff 



Coef. 

SE 

Sig. 

Treatment 

-0.12 

0.17 


Enrollment (in 1000s) 

-0.08 

0.14 


% FRP 

0.17 

0.30 


LEA ISP 

1.55 

0.77 

** 

Middle school grades sampled 

-0.23 

0.09 

** 

High school grades sampled 

-0.23 

0.13 

* 

Illinois 

-0.41 

0.32 


Kentucky 

-0.24 

0.32 


Michigan 

-0.25 

0.31 


New York 

-0.64 

0.34 

* 

Ohio 

-0.55 

0.39 


Constant 

0.22 

0.38 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.8: Administrative Time and Cost Regression Results for Hours per Student, School 
Foodservice Staff 



Coef. 

SE 

Sig. 

Treatment 

-0.31 

0.32 


Enrollment (in 1000s) 

-0.88 

0.39 

** 

% FRP 

-0.24 

0.69 


LEA ISP 

1.25 

1.53 


Middle school grades sampled 

-0.09 

0.27 


High school grades sampled 

0.23 

0.31 


Illinois 

1.73 

0.53 

*** 

Kentucky 

2.87 

0.48 

*** 

Michigan 

0.89 

0.38 

** 

New York 

1.33 

0.57 

** 

Ohio 

1.00 

0.82 


Constant 

0.53 

0.70 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<05, *= p<.10 


Source: Administrative Cost Interview and Staff Roster data 
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Exhibit 6A.9: Administrative Time and Cost Regression Results for Hours per Student, School 
non-Foodservice Staff 



Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.10: Administrative Time and Cost Regression Results for Hours per Student, 
School Staff, Role Unknown 



Coef. 

SE 

Sig. 

Treatment 

-0.09 

0.10 


Enrollment (in 1000s) 

-0.03 

0.08 


% FRP 

0.14 

0.20 


LEA ISP 

0.04 

0.18 


Middle school grades sampled 

0.01 

0.03 


High school grades sampled 

0.00 

0.03 


Illinois 

-0.62 

0.31 

** 

Kentucky 

-0.61 

0.30 

** 

Michigan 

-0.61 

0.31 

* 

New York 

-0.61 

0.31 

** 

Ohio 

-0.51 

0.31 


Constant 

0.55 

0.36 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 
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Exhibit 6A.11: Administrative Time and Cost Regression Results for Cost per Student, LEA 
Staff 



Coef. 

SE 

Sig. 

Treatment 

-7.45 

5.08 


Enrollment (in 1000s) 

-3.15 

4.30 


% FRP 

6.00 

11.71 


LEA ISP 

47.36 

25.92 

* 

Middle school grades sampled 

-5.11 

2.10 

** 

High school grades sampled 

-5.58 

3.10 

* 

Illinois 

-11.30 

10.55 


Kentucky 

-9.51 

9.82 


Michigan 

-11.67 

9.44 


New York 

-20.65 

10.68 

* 

Ohio 

-20.95 

11.93 

* 

Constant 

9.94 

11.15 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p< 05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.12: Administrative Time and Cost Regression Results for Cost per Student, School 
Foodservice Staff 



Coef. 

SE 

Sig. 

Treatment 

-8.11 

6.22 


Enrollment (in 1000s) 

-15.26 

8.13 

* 

% FRP 

-5.94 

15.08 


LEA ISP 

27.60 

32.63 


Middle school grades sampled 

-2.00 

5.07 


High school grades sampled 

4.28 

6.05 


Illinois 

36.77 

12.48 

*** 

Kentucky 

50.21 

10.21 

*** 

Michigan 

13.13 

7.98 


New York 

24.67 

10.20 

** 

Ohio 

18.13 

15.38 


Constant 

12.20 

14.54 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<05, *= p<.10 


Source: Administrative Cost Interview and Staff Roster data 
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Exhibit 6A.13: Administrative Time and Cost Regression Results for Cost per Student, School 
non-Foodservice Staff 



Coef. 

SE 

Sig. 

Treatment 

0.51 

5.86 


Enrollment (in 1000s) 

-13.95 

9.10 


% FRP 

35.05 

24.45 


LEA ISP 

-54.37 

36.63 


Middle school grades sampled 

-9.21 

3.74 

** 

High school grades sampled 

-2.55 

6.19 


Illinois 

-11.81 

16.45 


Kentucky 

-37.20 

14.20 

** 

Michigan 

-33.97 

13.31 

** 

New York 

-27.40 

12.71 

** 

Ohio 

-17.01 

18.11 


Constant 

51.11 

19.13 

*** 

Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.14: Administrative Time and Cost Regression Results for Cost per Student, School 
Staff, Role Unknown 



Coef. 

SE 

Sig. 

Treatment 

-1.68 

2.64 


Enrollment (in 1000s) 

-0.81 

2.22 


% FRP 

3.58 

5.12 


LEA ISP 

1.73 

6.39 


Middle school grades sampled 

0.61 

1.24 


High school grades sampled 

-0.12 

0.90 


Illinois 

-17.22 

8.09 

** 

Kentucky 

-16.84 

7.92 

** 

Michigan 

-17.05 

8.20 

** 

New York 

-17.13 

7.98 

** 

Ohio 

-12.82 

8.30 


Constant 

14.97 

10.49 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 
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Exhibit 6A.15: Administrative Time and Cost Regression Results for Hours per Student, 
Distributing and Processing Applications for Free or Reduced-Price Meals 



Coef. 

SE 

Sig. 

Treatment 

-0.34 

0.08 

*** 

Enrollment (in 1000s) 

-0.02 

0.08 


% FRP 

0.39 

0.31 


LEA ISP 

-0.45 

0.42 


Middle school grades sampled 

-0.03 

0.03 


High school grades sampled 

-0.01 

0.06 


Illinois 

0.19 

0.17 


Kentucky 

-0.08 

0.13 


Michigan 

-0.03 

0.10 


New York 

-0.08 

0.12 


Ohio 

0.15 

0.16 


Constant 

0.42 

0.13 

*** 

Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.16: Administrative Time and Cost Regression Results for Hours per Student, Direct 
Certifications 



Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 
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Exhibit 6A.17: Administrative Time and Cost Regression Results for Hours per Student, 
Verifying Income of Free/Reduced Price Students 



Coef. 

SE 

Sig. 

Treatment 

-0.11 

0.04 

*** 

Enrollment (in 1000s) 

-0.06 

0.05 


% FRP 

0.19 

0.15 


LEA ISP 

-0.14 

0.27 


Middle school grades sampled 

-0.05 

0.02 

kk 

High school grades sampled 

-0.01 

0.04 


Illinois 

0.14 

0.08 


Kentucky 

-0.01 

0.03 


Michigan 

0.02 

0.04 


New York 

0.08 

0.09 


Ohio 

0.03 

0.07 


Constant 

0.07 

0.11 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.18: Administrative Time and Cost Regression Results for Hours per Student, Meal 
Payment Collections and Accounting 



Coef. 

SE 

Sig. 

Treatment 

-0.68 

0.32 

** 

Enrollment (in 1000s) 

-1.12 

0.39 

*** 

% FRP 

-0.49 

0.65 


LEA ISP 

-0.50 

1.19 


Middle school grades sampled 

-0.28 

0.22 


High school grades sampled 

0.10 

0.28 


Illinois 

-0.14 

0.56 


Kentucky 

1.37 

0.64 

** 

Michigan 

-0.43 

0.50 


New York 

0.21 

0.65 


Ohio 

0.28 

0.86 


Constant 

3.06 

0.80 

•kick 

Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 
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Exhibit 6A.19: Administrative Time and Cost Regression Results for Hours per Student, 
Counting and Claiming Reimbursable Meals 



Coef. 

SE 

Sig. 

Treatment 

0.44 

0.16 

*** 

Enrollment (in 1000s) 

-0.33 

0.20 

* 

% FRP 

1.42 

0.66 

** 

LEA ISP 

1.17 

1.27 


Middle school grades sampled 

-0.29 

0.14 

** 

High school grades sampled 

-0.12 

0.16 


Illinois 

0.29 

0.38 


Kentucky 

-0.33 

0.19 

* 

Michigan 

-0.41 

0.26 


New York 

-0.67 

0.32 

** 

Ohio 

-0.54 

0.38 


Constant 

-0.46 

0.49 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.20: Administrative Time and Cost Regression Results for Cost per Student, 
Distributing and Processing Applications for Free or Reduced-Price Meals 



Coef. 

SE 

Sig. 

Treatment 

-10.84 

2.29 

*** 

Enrollment (in 1000s) 

-0.03 

2.19 


% FRP 

7.27 

6.76 


LEA ISP 

-2.64 

10.38 


Middle school grades sampled 

-0.42 

0.86 


High school grades sampled 

-0.77 

1.76 


Illinois 

2.66 

4.29 


Kentucky 

-3.12 

4.09 


Michigan 

-2.88 

3.20 


New York 

-4.53 

3.69 


Ohio 

0.68 

4.73 


Constant 

12.02 

3.97 

*** 

Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


pg. 6A-12 1 Appendix 6A 


Abt Associates 




































Community Eligibility Provision Evaluation: Final Report 


Exhibit 6A.21: Administrative Time and Cost Regression Results for Cost per Student, Direct 
Certifications 



Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.22: Administrative Time and Cost Regression Results for Cost per Student, 
Verifying Income of Free/Reduced Price Students 



Coef. 

SE 

Sig. 

Treatment 

-3.88 

1.31 

*** 

Enrollment (in 1000s) 

-1.14 

1.23 


% FRP 

3.47 

4.00 


LEA ISP 

0.63 

9.16 


Middle school grades sampled 

-1.33 

0.66 

** 

High school grades sampled 

-0.64 

1.26 


Illinois 

3.32 

2.72 


Kentucky 

-0.17 

0.96 


Michigan 

0.68 

1.37 


New York 

1.16 

2.66 


Ohio 

0.00 

2.75 


Constant 

1.82 

3.81 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 
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Exhibit 6A.23: Administrative Time and Cost Regression Results for Cost per Student, Meal 
Payment Collections and Accounting 



Coef. 

SE 

Sig. 

Treatment 

-14.10 

7.13 

* 

Enrollment (in 1000s) 

-23.73 

8.92 

*** 

% FRP 

-16.99 

16.20 


LEA ISP 

-7.91 

30.45 


Middle school grades sampled 

-6.46 

4.16 


High school grades sampled 

1.78 

6.22 


Illinois 

-10.34 

16.25 


Kentucky 

7.01 

17.30 


Michigan 

-26.23 

14.80 

* 

New York 

-10.88 

14.86 


Ohio 

-10.14 

18.75 


Constant 

82.86 

19.83 

*** 

Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<.05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 


Exhibit 6A.24: Administrative Time and Cost Regression Results for Cost per Student, 
Counting and Claiming Reimbursable Meals 



Coef. 

SE 

Sig. 

Treatment 

12.09 

3.85 

*** 

Enrollment (in 1000s) 

-7.69 

5.02 


% FRP 

33.86 

12.69 

*** 

LEA ISP 

29.62 

24.76 


Middle school grades sampled 

-7.06 

2.62 

*** 

High school grades sampled 

-2.95 

3.75 


Illinois 

-5.35 

9.75 


Kentucky 

-17.44 

6.56 

*** 

Michigan 

-19.48 

7.05 

*** 

New York 

-23.72 

8.60 

*** 

Ohio 

-20.17 

9.68 

** 

Constant 

-4.29 

10.93 


Number of schools 

247 



Number of LEAs 

100 




Reference categories are Elementary for grades sampled and West Virginia for state. 
***=p<.01, **=p<05, *= p<.10 

Source: Administrative Cost Interview and Staff Roster data 
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Appendix 7A: Analysis of Impacts on Administrative Errors in the 
Certification Process 


This appendix presents supporting information for the discussion in Chapter 7 about the impacts of 
the CEP on administrative errors in the certification process. As defined in the chapter, these errors 
include certification errors and procedural errors. The appendix begins with information on the data 
and results of balance testing. Next, univariate statistics for outcome variables are presented. The 
appendix concludes with the complete regression results supporting the impact estimates. 

7A.1 Data and Balance Testing 

The analysis file for the analysis of administrative errors in the certification process was constructed 
by merging data from certification records and FRP applications that were sampled and collected 
from schools within the Component 2 sample of treatment and comparison LEAs. The analysis file 
comprised data on 248 schools (119 treatment and 129 comparison) in 102 LEAs (51 treatment and 
51 comparison). A total of 9,257 records of identified students were sampled (5,401 from treatment 
schools and 3,856 from comparison schools). In addition, 2,920 FRP applications were sampled from 
comparison schools. Of these applications, 1,764 were approved and 1,156 were denied. 

Exhibit 7A. 1 repeats the balance analysis from Exhibit 2.7 from Chapter 2 for the 102 LEAs that 
participated in the certification record data collection. The substantive conclusions from Chapter 2 are 
unaltered: treatment and comparison LEAs for the certification process error analysis are balanced on 
all LEA-level observed covariates with the exception of ISP. Because the ISP was unbalanced 
between treatment and comparison LEAs, it was including as a control variable in the impact 
estimation model. 


Abt Associates 
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Exhibit 7A.1. LEA Characteristics for the Certification Process Error Analysis Sample 



Comparison 

Treatment 



Mean (SD) 

Mean (SD) 

Enrollment 

3,854 

(5752) 

4,013 

(7016) 

Enrollment 1-499 (%) 

17.65 


21.57 


Enrollment 500-2,499 (%) 

45.1 


37.25 


Enrollment 2,500-4,999 (%) 

13.73 


25.49 


Enrollment 5,000+ (%) 

23.53 


15.69 


Percentage distribution of students in grades K-12 

Percent in grades K-5 

0.5 

(0.2) 

0.51 

(0.1) 

Percent in grades 6-8 

0.21 

(0.1) 

0.24 

(0.1) 

Percent in grades 9-12 

0.26 

(0.2) 

0.22 

(0.1) 

Number of schools 

7.47 

(8.8) 

8.73 

(12.9) 

1 school (%) 

17.65 


17.65 


2-5 schools (%) 

37.25 


37.25 


6-14 schools (%) 

35.29 


27.45 


15+ schools (%) 

9.8 


11.76 


ISP 

47.92 

(16.2)** 

54.85 

(16.5)** 

Percent students approved for free/reduced 

70.58 

(15.2) 

74.09 

(14.6) 

Percent Title 1 schools 

86.41 

(21.3) 

87.15 

(15.3) 

Urban LEA (%) 

27.45 


29.41 


All charter schools (%) 

15.69 


19.61 


Percent English Language Learners 

3.1 

(5.1) 

3.38 

(7.0) 

Percent students Black 

26.95 

(31.3) 

32.85 

(35.9) 

Percent students Hispanic/Latino 

7.99 

(11.6) 

8.84 

(15.1) 

Average Daily Participation (%) Lunch 

68.2 

(11.5) 

69.84 

(12.9) 

Number of LEAs 

51 


51 



T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade distribution. Differences in grade distributions were jointly tested using MANOVA. 

Source: Common Core of School Data, State Administrative Data, Verification Summary Report Data 
***=p<0.01; **=p<0.05; *=p<0.10 


Data were collected from 248 schools in the 102 LEAs that participated in the certification records 
data collection. Characteristics of these schools are shown in Exhibit 7A.2. Treatment schools had 
somewhat lower average enrollment then comparison schools (478 versus 576) and more students 
eligible for free and reduced-price meals (70.2 versus 66.2 percent). Treatment and comparison 
schools were otherwise statistically indistinguishable. Because enrollment and the percentage of 
students eligible for free and reduced-price meals were unbalanced between treatment and 
comparison schools, they were include as a control variables in the impact estimation model. 
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Exhibit 7A.2. School Characteristics for the Certification Process Error Analysis Sample 



Number of schools 129 119 


a The grade span refers to the actual grade coverage of a school. Grade coverage can be irregular, especially 
among charter schools, which may cover K-12, K-8, or other grade spans, all of which are included in the “Other” 
category in the table. The analysis has categorized these irregular grade spans according to three separate 
sampled grades: elementary (K-5), middle (6-8), and high (9-12). The table presents both the original grade span 
and the sampled grade category. 

T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade span and grades sampled. Differences in grade span and grades sampled were jointly tested using a chi- 
square test. 

Source: Common Core of School Data 
***=p<.01, **=p<.05, *=p<.10 


7A.2 Univariate Statistics for Outcome Variables 

Exhibit 7A.3 shows the means and standard deviation for the outcome variables used in the analysis 
of CEP impacts on procedural error rates for identified students. 


Exhibit 7A.3. Procedural Error Rates for Identified Students 



Comparison Schools 

Treatment Schools 


Mean 

(SD) 

Mean 

(SD) 

No supporting documentation for free-eligible 
without application 

3.0 

(9.6) 

2.1 

(7.3) 

Mismatched identifiers between FRP certification 
lists and supporting documents 

1.2 

(4.0) 

1.0 

(3.3) 

Any procedural error 

4.3 

(10.6) 

3.1 

(8.4) 

Number of schools 

129 


119 



T-tests were used to test for differences between treatment and comparison means for all outcomes 

***=p<0.01; **=p<0.05; *=p<0.10 

Source: Certification Record Review Data Collection 
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7A.3 Regression Results 


Exhibits 7A.4-7A.6 present the complete set of regression results for CEP impacts on procedural and 
certification error rates for identified students. The vector of school characteristics used as control 
variables in all models included: 

• the school’s grade level (i.e., whether elementary, middle, or high), with high school as the 
reference category; 

• the State, with West Virginia as the reference State; and 

• LEA and school-level features where balance testing (detailed above) discerned a difference 
between treatment and comparison LEAs that was statistically significant at the 10 percent 
level, which included school enrollment and FRP participation, as well as the LEA’s ISP. 


Exhibit 7A.4. Procedural Error for Identified Students: No Supporting Documentation for Free- 
Eligible without Application 



Reference categories are Elementary for grades sampled and West Virginia for state. 

***=p<0.01; **=p<0.05; *=p<0.10 

Source: Certification Record Review Data Collection 
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Exhibit 7A.5. Procedural Error for Identified Students: Mismatched Identifiers between FRP 
Certification Lists and Supporting Documents 



Reference categories are Elementary for grades sampled and West Virginia for state. 

***=p<0.01; **=p<0.05; *=p<0.10 

Source: Certification Record Review Data Collection 


Exhibit 7A.6. Procedural Error for Identified Students: Any Procedural Error 



Coef. 

SE 

Sig. 

Treatment 

-1.03 

1.77 


Enrollment (in 1000s) 

4.96 

2.90 


% FRP 

-4.57 

4.74 

* 

LEA ISP 

4.67 

7.08 


Middle school grades sampled 

-0.26 

0.84 


High school grades sampled 

-2.29 

1.54 


Illinois 

5.66 

2.75 


Kentucky 

3.29 

1.61 

** 

Michigan 

1.48 

1.41 

** 

New York 

5.60 

5.08 


Ohio 

-0.93 

2.00 


Constant 

0.56 

3.05 


Number of observations 

245 



Number of LEAs 

101 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***=p<0.01; **=p<0.05; *=p<0.10 

Source: Certification Record Review Data Collection 


Abt Associates 
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Appendix 7B: Analysis of Impacts on Cashier Errors 


This appendix presents supporting information for the discussion in Chapter 7 about the impacts of 
the CEP on cashier errors in identifying reimbursable meals. The appendix begins with information 
on the data and results of balance testing. Next, univariate statistics for outcome variables are 
presented. The appendix concludes with the complete regression results supporting the impact 
estimates. 


7B.1 Data and Balance Testing 

7B.1.1 Balance Testing at LEA Level (Same for NSLP and SBP) 

The analysis file for the cashier observation was constructed using data from the cashier observations 
data and the meal survey interview, completed as part of the data collection for Component 3 of the 
Impact Study, as described in Chapter 2. Two comparison LEAs selected for this data collection were 
found ineligible to participate. Therefore, the sample was drawn from 52 LEAs (27 treatment and 25 
comparison) as explained in Chapter 2. Exhibit 7B.1 repeats the analysis from Exhibit 2.7 from 
Chapter 2 for this set of LEAs. The substantive conclusions from Chapter 2 are unaltered: treatment 
and comparison LEAs for the cashier observation analysis are balanced on all LEA-level observed 
co variates. 


Abt Associates 
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Exhibit 7B.1: LEA Characteristics for Cashier Observations Analysis Sample 



Component 3 (Cashier Observations) 


Comparison 

Treatment 


Mean (SD) 

Mean (SD) 

Number of observations 

25 


27 


Enrollment 

5546 

(7539) 

6803 

(8806) 

Enrollment 1-499 (%) 

0 


3.7 


Enrollment 500-2,499 (%) 

48 


29.6 


Enrollment 2,500-4,999 (%) 

20 


33.3 


Enrollment 5,000+ (%) 

32 


33.3 


Percentage distribution of students in grades K-12 

Percent in grades K-5 

44 

(4.4) 

44.3 

(3.4) 

Percent in grades 6-8 

21.9 

(1.7) 

21.8 

(1.7) 

Percent in grades 9-12 

30.1 

(5.5) 

29.5 

(3.3) 

Number of schools 

10.7 

(11.1) 

14.4 

(15.8) 

1 school (%) 

0 


0 


2-5 schools (%) 

36 


29.6 


6-14 schools (%) 

44 


37 


15+ schools (%) 

20 


25.9 


ISP 

44.5 

(11.1) 

49.6 

(12.3) 

Percent students free/reduced lunch 

66.2 

(14.2) 

70.3 

(12.7) 

Percent Title 1 schools 

79.1 

(25.6) 

80.4 

(14.4) 

Urban LEA (%) 

20 


33.3 


All charter schools (%) 

0 


3.7 


Percent English Language Learners 

3.2 

(4.4) 

4.5 

(8.2) 

Percent students Black 

15.5 

(19.5) 

24.2 

(28.5) 

Percent students Hispanic/Latino 

7.6 

(10.5) 

10.5 

(17.8) 

Average Daily Participation (%) Lunch 

65.6 

(10.5) 

67.2 

(15.1) 


T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade distribution. Differences in grade distributions were jointly tested using MANOVA. 

***=p<0.01; **=p<0.05; *=p<0.10 

Source: Common Core of School Data, State Administrative Data, Verification Summary Report Data 

7B.1.2 Balance Testing at School Level 
National School Lunch Program 

Data were collected from 155 schools in the 54 LEAs that agreed to participate in Component 3 data 
collection for cashier observations for lunch. Characteristics of these schools are shown in Exhibit 
7B.2. Treatment schools had more students eligible for free and reduced lunch and a higher 
percentage of students who were Black. Treatment schools also tend to be more urban, Title I or 
charter schools. On all over characteristics, the two groups of schools were statistically 
indistinguishable. The characteristics which were unbalanced between the treatment and comparison 
schools were included as control variables in the impact estimation model for lunch. 
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Exhibit 7B.2: Cashier Observations, NSLP Sample, School Level Measures 



Percent students Hispanic/Latino 6.9 (10.0) 10.4 (17.8) 


a The grade span refers to the actual grade coverage of a school. Grade coverage can be irregular, especially 
among charter schools, which may cover K-12, K-8, or other grade spans, all of which are included in the “Other” 
category in the table. The analysis has categorized these irregular grade spans according to three separate 
sampled grades: elementary (K-5), middle (6-8), and high (9-12). The table presents both the original grade span 
and the sampled grade category. 

T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade span and grades sampled. Differences in grade span and grades sampled were jointly tested using a chi- 
square test. 

***=p<0.01; **=p<0.05; *=p<0.10 
Source: Common Core of School Data 

School Breakfast Program 

Data were collected from 148 schools in the 54 LEAs that agreed to participate in Component 3 data 
collection for cashier observations for breakfast. Characteristics of these schools are shown in Exhibit 
7B.3. Treatment schools had more students eligible for free and reduced lunch and a higher 
percentage of students who were Black. Treatment schools also tend to be more urban and charter 
schools. On all over characteristics, the two groups of schools were statistically indistinguishable. The 
characteristics which were unbalanced between the treatment and comparison schools were included 
as control variables in the impact estimation model for breakfast. 
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Exhibit 7B.3: Cashier Observations, SBP Sample, School Level Measures 



Percent students Hispanic/Latino 7.1 (10.1) 10.2 (17.9) 


a The grade span refers to the actual grade coverage of a school. Grade coverage can be irregular, especially 
among charter schools, which may cover K-12, K-8, or other grade spans, all of which are included in the “Other” 
category in the table. The analysis has categorized these irregular grade spans according to three separate 
sampled grades: elementary (K-5), middle (6-8), and high (9-12). The table presents both the original grade span 
and the sampled grade category. 

T-tests were used to test for differences between treatment and comparison means for all variables 
except for grade span and grades sampled. Differences in grade span and grades sampled were 
jointly tested using a chi-square test. 

***=p<0.01; **=p<0.05; *=p<0.10 
Source: Common Core of School Data 


7B.2 Univariate statistics for outcome variables 

Exhibit 7B.4 shows the means and standard deviation for the types of error used to estimate the 
outcome which was used in the analysis of CEP impacts on cashier observations for breakfast and 
lunch. 

The majority of cashier errors comes from over-claiming, where the cashier is counting meals as 
reimbursable when they are not. Most of the over-claiming error is due to students taking meals with 
missing components which would make the meal not reimbursable. The cashier error rate for 
counting breakfast meals for the schools that are participating in the CEP is about five percent, which 
is lower than the error rate for non-participating schools (6.4 percent). A similar pattern is seen in 
cashier errors for counting lunches but the error rate is much lower, 2.8 percent for participating 
schools and 3.7 percent for non-participating schools. The error rates found for this study are similar 
to those reported error rates found in the APEC study which gives confidence to the results which are 
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findings presented in the Results Section. 1 Exhibit 7B.5 does the same across grades for breakfast 
and lunch separately. 

Exhibit 7B.4: Cashier Error Rates in School Breakfast and Lunch Program by Type of Error 


Breakfast Lunch 



Comparison 

Treatment 

Comparison 

Treatment 

Type of Error 

Mean 

Std 

Dev 

Mean 

Std 

Dev 

Mean 

Std 

Dev 

Mean 

Std 

Dev 

Underclaiming 

0.35% 

1.37% 

0.20% 

1.30% 

0.31% 

1.73% 

0.11% 

0.49% 

error 









Overclaiming error 

6.05 

7.82 

4.94 

8.18 

3.41 

6.32 

2.69 

7.03 

Total cashier error 

6.40 

8.17 

5.14 

8.39 

3.72 

6.55 

2.80 

7.05 

Number of 

73 

75 

75 

80 

schools 









Number of LEAs 

25 

27 

25 

27 

Notes: Under-claiming Error is the number of meals not counted as reimbursable when they are divided by the 

total number of meals served. Over-claiming Error is the number of meals counted as reimbursable when they 

are not divided by the total number of meals served. The Total Cashier Error rate is the sum of the over-claiming 

and under-claiming error rate. *** p<0.01; **p<0.05; * p<0.10 





Source: Cashier Observations and Menu Survey Data Collection 





Exhibit 7B.5: Cashier Error Rate across Grades for Breakfast and Lunch 





Breakfast 



Lunch 



Comparison 

Treatment 

Comparison 

Treatment 



Std 


Std 


Std 


Std 

Type of Error 

Mean 

Dev 

Mean 

Dev 

Mean 

Dev 

Mean 

Dev 

Elementary schools 

Underclaiming error 

0.42% 

1.53% 

0.00% 

0.00% 

0.66% 

2.73% 

0.00% 

0.00% 

Overclaiming error 

6.33%“ 

10.15% 

1.62%“ 

4.73% 

1.39% 

1.81% 

3.48% 

10.05% 

Total cashier error 

6.75%“ 

10.45% 

1.62%“ 

4.73% 

2.05% 

3.76% 

3.48% 

10.05% 

Number of schools 

26 

29 

26 

31 

Middle schools 





Underclaiming error 

0.19% 

0.95% 

0.00% 

0.00% 

0.22% 

1.11% 

0.27% 

0.81% 

Overclaiming error 

5.60% 

6.24% 

5.72% 

7.38% 

1.93% 

3.25% 

2.54% 

4.19% 

Total cashier error 

5.80% 

6.72% 

5.72% 

7.38% 

2.15% 

3.30% 

2.81% 

4.31% 

Number of schools 

24 

22 

25 

25 

High schools 

Underclaiming error 

0.42% 

1.57% 

0.64% 

2.26% 

0.03% 

0.14% 

0.07% 

0.33% 

Overclaiming error 

6.20% 

1.36% 

8.23% 

2.17% 

7.13%“ 

9.61% 

1.84%“ 

4.23% 

Total cashier error 

6.62% 

4.31% 

8.87% 

10.92% 

7.15%“ 

9.66% 

1.91%“ 

4.23% 

Number of schools 

23 

24 

24 

24 


Notes: Under-claiming Error is the number of meals not counted as reimbursable when they are divided by the 
total number of meals served. Over-claiming Error is the number of meals counted as reimbursable when they 
are not divided by the total number of meals served. The Total Cashier Error rate is the sum of the over-claiming 
and under-claiming error rate. *** p<0.01; **p<0.05; * p<0.10 
Source: Cashier Observations and Menu Survey Data Collection 


1 For a benchmark, the APEC study found that total gross erroneous payments from cashier error represented 
3.1 percent of total reimbursements in the NSLP and 9.8 percent of total SBP reimbursements. (Ponza, M., 
et al., p. 12) 
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7B.3 Regression Results 

Exhibits 7B.6-8 present the complete set of regression results for lunch and breakfast respectively. 

The vector of school characteristics used as control variables in all models included: 

• the school’s grade level (i.e., whether elementary, middle, or high), with high school as the 
reference category; 

• the State, with West Virginia as the reference State; and 

• LEA and school-level features where balance testing (detailed above) discerned a difference 
between treatment and comparison LEAs that was statistically significant at the 10 percent level, 
which included school free and reduced price percentage, school is urban, Title I or charter school 
and percentage of Black students in the school. 

Exhibit 7B.6: Cashier Error Rate in School Lunch Program Regression Results for All Grades 

Combined 


CashierErr 

Coef. 

Std. Err. 

Significance 

Treatment 

-0.008 

0.011 


High school indicator 

0.015 

0.015 


Middle school indicator 

-0.005 

0.012 


Students free/reduced price (%) 

0.058 

0.039 


Urban LEA 

0.010 

0.018 


Title 1 school 

-0.025 

0.019 


Charter school 

-0.055 

0.028 

* 

Percent students Black 

0.001 

0.034 


IL state indictor 

-0.003 

0.021 


KY state indictor 

0.028 

0.017 


Ml state indictor 

0.032 

0.021 


NY state indictor 

0.013 

0.017 


OH state indictor 

-0.018 

0.016 


Intercept 

0.000 

0.025 


Number of LEAs 

52 



Number of schools 

155 




Note: Reference for grade is Elementary. Reference for state is West Virginia state indictor. 

Only school level variables that are not balanced are added due to the number of observation limitations. 
***=p<0.01; **=p<0.05; *=p<0.10 

Source: Cashier Observations and Menu Survey Data Collection 
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Exhibit 7B.7: Cashier Error Rate in School Lunch Program Regression Results by Grade Level 



Note: Reference for grade is Elementary. Reference for state is West Virginia state indictor. 

Only school level variables that are not balanced are added due to the number of observation limitations. 
***=p<0.01; **=p<0.05; *=p<0.10 

Source: Cashier Observations and Menu Survey Data Collection 

Exhibit 7B.8: Cashier Error Rate in School Breakfast Program Regression Results for All 
Schools Combined 


CashierErr 

Coef. 

Std. Err. 

Significance 

Treatment 

-0.011 

0.014 


High school indicator 

0.019 

0.023 


Middle school indicator 

0.008 

0.017 


Students free/reduced price (%) 

-0.031 

0.067 


Urban LEA 

-0.013 

0.019 


Title 1 school 

-0.033 

0.026 


Charter school 

0.086 

0.032 

*** 

Percent students Black 

0.044 

0.035 


IL state indictor 

-0.002 

0.018 


KY state indictor 

0.043 

0.018 

** 

Ml state indictor 

0.062 

0.029 

** 

NY state indictor 

0.046 

0.028 


OH state indictor 

0.022 

0.039 


Intercept 

0.065 

0.047 


Number of LEAs 

52 



Number of schools 

148 




Note: Reference for grade is Elementary. Reference for state is West Virginia state indictor. 

Only school level variables that are not balanced are added due to the number of observation limitations. 
***=p<0.01; **=p<0.05; *=p<0.10 

Source: Cashier Observations and Menu Survey Data Collection 
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Exhibit 7B.9: Cashier Error Rate in School Breakfast Program Regression Results by Grade 
Level 


CashierErr 

Coef. 

Std. Err. 

Significance 

High school indicator 

-0.015 

0.027 


Middle school indicator 

-0.017 

0.024 


Treatment impact at elementary school 

-0.049 

0.020 

** 

Treatment impact at high school 

0.018 

0.024 


Treatment impact at middle school 

0.001 

0.021 


Students free/reduced price (%) 

-0.019 

0.068 


Urban LEA 

-0.011 

0.020 


Title 1 school 

-0.034 

0.026 


Charter school 

0.086 

0.032 

*** 

Percent students Black 

0.038 

0.036 


IL state indictor 

-0.002 

0.018 


KY state indictor 

0.043 

0.019 

** 

Ml state indictor 

0.059 

0.030 

* 

NY state indictor 

0.045 

0.029 


OH state indictor 

0.017 

0.042 


Intercept 

0.078 

0.047 


Number of LEAs 

52 



Number of schools 

148 




Note: Reference for grade is Elementary. Reference for state is West Virginia state indictor. 

Only school level variables that are not balanced are added due to the number of observation limitations. 
***=p<0.01; **=p<0.05; *=p<0.10 

Source: Cashier Observations and Menu Survey Data Collection 
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Appendix 7C: Analysis of Impacts on Meal Claiming Errors 


This appendix presents supporting information for the discussion in Chapter 7 about the impacts of 
the CEP on meal claiming errors at the school and LEA levels. The appendix begins with information 
on the data and results of balance testing. Next, univariate statistics for outcome variables are 
presented. The appendix concludes with the complete regression results supporting the impact 
estimates. 


7C.1 Data and Balance Testing 

Exhibit 1C. 1 presents the balance tests for the 47 LEAs included in the claiming analysis. All LEA- 
level observed covariates balance with the exception of the number of schools. Because the number 
of schools was unbalanced between treatment and comparison LEAs, it was including as a control 
variable in the impact estimation model. 


Exhibit 7C.1: LEA Characteristics for the Claiming Error Analysis Sample 



Comparison 

Mean (SD) 

Treatment 

Mean (SD) 

Enrollment 

4201 

(3484) 

7331 

(9429) 

Enrollment 1-499 (%) 

0 


4.3 


Enrollment 500-2,499 (%) 

50 


30.4 


Enrollment 2,500-4,999 (%) 

20.8 


30.4 


Enrollment 5,000+ (%) 

29.2 


34.8 


Percentage distribution of students in grades K-12 

Percent in grades K-5 

43.8 

(4.4) 

44.4 

(3.6) 

Percent in grades 6-8 

21.9 

(1.8) 

21.6 

(1.8) 

Percent in grades 9-12 

30.2 

(5.6) 

29.4 

(3.5) 

Number of schools 

8.9 

(6.6)* 

15.3 

(16.9)* 

1 school (%) 

0 


0 


2-5 schools (%) 

37.5 


30.4 


6-14 schools (%) 

45.8 


34.8 


15+ schools (%) 

16.7 


26.1 


ISP 

45.3 

(10.6) 

49.6 

(12.0) 

Percent students free/reduced lunch 

67 

(13.9) 

69.6 

(13.0) 

Percent Title 1 schools 

80.6 

(25.1) 

80.4 

(14.1) 

Urban LEA (%) 

16.7 


34.8 


All charter schools (%) 

0 


4.3 


Percent English Language Learners 

3 

(4.4) 

5.1 

(8.7) 

Percent students Black 

15.1 

(19.8) 

25.4 

(28.3) 

Percent students Hispanic/Latino 

7.5 

(10.7) 

11.8 

(19.0) 

Average Daily Participation (%) Lunch 

65.7 

(10.8) 

64.3 

(15.2) 

Number of LEAs 

24 


23 



T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade distribution. Differences in grade distributions were jointly tested using MANOVA. 

***p<0.01; **p<0.05; * p<0.10 

Source: Common Core of School Data, State Administrative Data, Verification Summary Report Data 

Characteristics of the 137 school included in the claiming error analysis are shown in Exhibit 7C.2. 

Treatment and comparison schools are statistically different on a number of features. Treatment 
schools are much more likely to be located in an urban area (36 versus 16 percent); about 4.5 percent 
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of treatment schools are charter schools, but no comparison schools are. Treatment schools have 
more Black students (26 versus 16 percent, on average), Hispanic/Latino students (12 versus 7 
percent, on average), and FRP meal eligible students (69 versus 64 percent, on average). Controls for 
these differences were included in the impact estimation model. 


Exhibit 7C.2: School Characteristics for the Claiming Error Analysis Sample 



Comparison 

Mean (SD) 

Treatment 

Mean (SD) 

Number of schools 

70 


67 


Enrollment 

591 

(452) 

561 

(370) 

Grade span 3 

K-5 (%) 

35.7 


32.8 


6-8 (%) 

30 


26.9 


9-12 (%) 

31.4 


31.3 


Other (%) 

2.9 


9 


Grades Sampled 3 

K-5 (%) 

35.7 


37.3 


6-8 (%) 

32.9 


29.9 


9-12 (%) 

31.4 


32.8 


Percent students free/reduced lunch 

64 

(14.8)** 

68.9 

(14.5)** 

Percent Title 1 schools 

75.7 

(43.2) 

86.6 

(34.4) 

Charter school (%) 

0 

(0.0)* 

4.5 

(20.8)* 

Urban School (%) 

15.7 

(36.7)*** 

35.8 

(48.3)*** 

Percent students Black 

15.6 

(21.2)** 

25.5 

(29.1)** 

Percent students Hispanic/Latino 

7 

(10.3)* 

11.8 

(19.0)* 


aThe grade span refers to the actual grade coverage of a school. Grade coverage can be irregular, especially 
among charter schools, which may cover K-12, K-8, or other grade spans, all of which are included in the “Other” 
category in the table. The analysis has categorized these irregular grade spans according to three separate 
sampled grades: elementary (K-5), middle (6-8), and high (9-12). The table presents both the original grade span 
and the sampled grade category. 

T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade span and grades sampled. Differences in grade span and grades sampled were jointly tested using a chi- 
square test. 

***p<0.01; **p<0.05; * p<0.10 
Source: Common Core of School Data 


7C.2 Univariate statistics for outcome variables 

Exhibits 7C.3 and 7C.4 show the means and standard deviation for the outcome variables used in the 
analysis of CEP impacts on claiming error rates. Exhibit 7C.3 shows the signed error rates, and 
Exhibit 7C.4 shows the unsigned error rates. 


pg.7C-2| Appendix 7C 


Abt Associates 

























Community Eligibility Provision Evaluation: Final Report 


Exhibit 7C.3: Signed Claiming Error Rate in NSLP, SBP, and Combined Programs 



Comparison 

Mean (SD) 

Treatment 

Mean (SD) 

NLSP signed claiming error rate 

Elementary schools 

-0.01 

(0.10) 

-1.51 

(8.76) 

Middle schools 

1.79 

(8.59) 

-2.53 

(11.46) 

High schools 

0.06 

(0.26) 

-1.42 

(13.09) 

All schools 

0.60 

(4.93) 

-1.78 

(10.98) 

SBP signed claiming error rate 

Elementary schools 

0.01 

(0.03) 

-1.07 

(8.22) 

Middle schools 

0.13 

(0.45) 

-2.28 

(10.11) 

High schools 

0.04 

(0.17) 

-1.76 

(11.28) 

All schools 

0.06 

(0.28) 

-1.66 

(9.73) 

Combined signed claiming error rate 

Elementary schools 

-0.01 

(0.07) 

-1.37 

(8.63) 

Middle schools 

1.62 

(7.71) 

-2.50 

(11.28) 

High schools 

0.05 

(0.23) 

-1.46 

(12.44) 

All schools 

0.55 

(4.42) 

-1.74 

(10.64) 

Number of schools 

Elementary schools 

25 


25 


Middle schools 

23 


20 


High schools 

22 


22 


All schools 

70 


67 


Number of LEAs 

24 


23 



T-tests were used to test for differences between treatment and comparison means for all outcomes 

***p<0.01; **p<0.05; * p<0.10 

Source: Counting and Claiming Error Data Collection 
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Exhibit 7C.4: Unsigned Claiming Error Rate in NSLP, SBP, and Combined Programs 



Comparison 

Mean (SD) 

Treatment 

Mean (SD) 

NLSP signed claiming error rate 

Elementary schools 

0.02 

(0.10) 

2.27 

(8.59) 

Middle schools 

1.79 

(8.59) 

2.59 

(11.44) 

High schools 

0.06 

(0.26) 

4.27 

(12.42) 

All schools 

0.61 

(4.92) 

3.02 

(10.70)* 

SBP signed claiming error rate 

Elementary schools 

0.01 

(0.03) 

2.38 

(7.93) 

Middle schools 

0.13 

(0.45) 

2.28 

(10.11) 

High schools 

0.04 

(0.17) 

4.35 

(10.52)* 

All schools 

0.06 

(0.28) 

3.00 

(9.40)*** 

Combined signed claiming error rate 

Elementary schools 

0.02 

(0.07) 

2.33 

(8.42) 

Middle schools 

1.62 

(7.71) 

2.55 

(11.27) 

High schools 

0.05 

(0.23) 

4.23 

(11.75) 

All schools 

0.56 

(4.42) 

3.02 

(10.34)* 

Number of schools 

Elementary schools 

25 


25 


Middle schools 

23 


20 


High schools 

22 


22 


All schools 

70 


67 


Number of LEAs 

24 


23 



T-tests were used to test for differences between treatment and comparison means for all outcomes 

***p<0.01; **p<0.05; * p<0.10 

Source: Counting and Claiming Error Data Collection 


7C.3 Regression Results 

Exhibits 7C.5-15 present the complete set of regression results. Exhibits 7C.5-10 present the 
regressions for the combined (all grade levels) impacts, while Exhibits 7C.11-15 present the 
regressions used for the grade level impact estimates. The vector of school characteristics used as 
control variables in all models included: 

• the school’s grade level (i.e., whether elementary, middle, or high), with elementary school as the 
reference category; 

• the State, with West Virginia as the reference State; and 

• LEA and school-level features where balance testing (detailed above) discerned a difference 
between treatment and comparison LEAs that was statistically significant at the 10 percent level, 
which included the LEA’s number of schools, whether the school was urban, whether the school 
was a charter school, the school’s enrollment, and the school’s percent Black, percent Hispanic, 
and percent FRP. 
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Exhibit 7C.5: Regression Results for Signed Claiming Error Rate in SBP, Overall 


Coef. 

SE 

Sig. 

Treatment 

-1.44 

0.77 

* 

Urban school 

0.70 

0.64 


Charter schools 

2.82 

1.52 

* 

Number of schools in LEA 

0.99 

2.12 


% FRP students 

-2.20 

1.92 


% Black students 

2.79 

5.14 


% Hispanic/Latino students 

-0.13 

0.09 


Middle school grades sampled 

-0.73 

1.39 


High school grades sampled 

-0.14 

2.12 


Illinois 

-1.66 

1.25 


Kentucky 

-4.15 

2.28 

* 

Michigan 

-2.22 

1.67 


New York 

-2.76 

1.56 

* 

Ohio 

-2.17 

2.02 


Constant 

1.11 

2.32 


Number of observations 

137 



Number of LEAs 

47 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 


Exhibit 7C.6: Regression Results for Unsigned Claiming Error Rate in SBP, Overall 



Coef. 

SE 

Sig. 

Treatment 

3.50 

1.62 

** 

Urban school 

-1.64 

1.09 


Charter schools 

-4.07 

2.45 


Number of schools in LEA 

-5.23 

5.27 


% FRP students 

0.79 

2.43 


% Black students 

-7.61 

5.86 


% Hispanic/Latino students 

0.11 

0.09 


Middle school grades sampled 

0.25 

1.39 


High school grades sampled 

0.27 

1.51 


Illinois 

2.77 

1.89 


Kentucky 

5.63 

2.23 

** 

Michigan 

6.87 

3.84 

* 

New York 

5.79 

2.98 

* 

Ohio 

3.98 

2.60 


Constant 

0.99 

2.75 


Number of observations 

137 



Number of LEAs 

47 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 


Abt Associates 


Appendix 7C | pg. 7C-5 










































Community Eligibility Provision Evaluation: Final Report 


Exhibit 7C.7: Regression Results for Signed Claiming Error Rate in NLSP, Overall 



Number of LEAs 47 


Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 
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Exhibit 7C.8: Regression Results for Unsigned Claiming Error Rate in NLSP, Overall 


Coef. 

SE 

Sig. 

Treatment 

3.00 

1.93 


Urban school 

-0.31 

1.82 


Charter schools 

-5.01 

2.91 

* 

Number of schools in LEA 

-6.81 

5.88 


% FRP students 

0.82 

2.86 


% Black students 

-8.42 

6.71 


% Hispanic/Latino students 

0.10 

0.11 


Middle school grades sampled 

1.35 

1.90 


High school grades sampled 

0.22 

1.78 


Illinois 

2.87 

2.03 


Kentucky 

6.03 

2.78 

** 

Michigan 

6.96 

4.32 


New York 

7.13 

3.41 

** 

Ohio 

3.68 

3.06 


Constant 

1.48 

3.24 


Number of observations 

137 



Number of LEAs 

47 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 


Exhibit 7C.9: Regression Results for Total Signed Claiming Error Rate, Overall 



Coef. 

SE 

Sig. 

Treatment 

-2.04 

1.09 

* 

Urban school 

2.36 

1.65 


Charter schools 

1.88 

1.84 


Number of schools in LEA 

-0.14 

2.63 


% FRP students 

-2.78 

2.33 


% Black students 

4.74 

5.99 


% Hispanic/Latino students 

-0.16 

0.11 


Middle school grades sampled 

0.16 

1.76 


High school grades sampled 

0.37 

2.30 


Illinois 

-1.98 

1.50 


Kentucky 

-4.73 

2.76 

* 

Michigan 

-3.17 

1.98 


New York 

-1.91 

2.08 


Ohio 

-3.29 

2.37 


Constant 

0.34 

2.56 


Number of observations 

137 



Number of LEAs 

47 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 
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Exhibit 7C.10: Regression Results for Total Unsigned Claiming Error Rate, Overall 


Coef. 

SE 

Sig. 

Treatment 

3.04 

1.85 


Urban school 

-0.46 

1.69 


Charter schools 

-4.86 

2.81 

* 

Number of schools in LEA 

-6.44 

5.73 


% FRP students 

0.75 

2.78 


% Black students 

-8.25 

6.56 


% Hispanic/Latino students 

0.10 

0.10 


Middle school grades sampled 

1.20 

1.81 


High school grades sampled 

0.18 

1.69 


Illinois 

2.83 

2.00 


Kentucky 

5.91 

2.63 

** 

Michigan 

6.93 

4.21 


New York 

6.94 

3.31 

** 

Ohio 

3.68 

2.97 


Constant 

1.40 

3.12 


Number of observations 

137 



Number of LEAs 

47 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 


Exhibit 7C.11: Regression Results for Signed Claiming Error Rate in SBP, Grade-Level 
Differences 



Coef. 

SE 

Sig. 

Treatment 

-0.57 

2.01 


Treatment * Middle school 

-1.80 

2.99 


Treatment * High school 

-0.92 

4.01 


Urban school 

0.64 

0.62 


Charter schools 

2.97 

1.67 

* 

Number of schools in LEA 

0.95 

2.13 


% FRP students 

-2.16 

1.85 


% Black students 

2.46 

4.93 


% Hispanic/Latino students 

-0.13 

0.09 


Middle school grades sampled 

0.11 

0.11 


High school grades sampled 

0.29 

0.46 


Illinois 

-1.65 

1.26 


Kentucky 

-4.22 

2.37 

* 

Michigan 

-2.14 

1.62 


New York 

-2.72 

1.53 

* 

Ohio 

-2.02 

1.97 


Constant 

0.88 

2.12 


Number of observations 

137 



Number of LEAs 

47 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 
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Exhibit 7C.12: Regression Results for Unsigned Claiming Error Rate in SBP, Grade-Level 
Differences 



Coef. 

SE 

Sig. 

Treatment 

2.90 

2.40 


Treatment * Middle school 

-0.17 

3.19 


Treatment * High school 

2.04 

2.47 


Urban school 

-1.69 

1.10 


Charter schools 

-4.20 

2.56 


Number of schools in LEA 

-5.29 

5.37 


% FRP students 

0.85 

2.40 


% Black students 

-7.48 

5.94 


% Hispanic/Latino students 

0.11 

0.09 


Middle school grades sampled 

0.30 

0.37 


High school grades sampled 

-0.75 

0.66 


Illinois 

2.79 

1.92 


Kentucky 

5.53 

2.30 

** 

Michigan 

6.89 

3.92 

* 

New York 

5.82 

3.02 

* 

Ohio 

3.81 

2.57 


Constant 

1.23 

2.61 


Number of observations 

137 



Number of LEAs 

47 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 


Exhibit 7C.13: Regression Results for Signed Claiming Error Rate in NLSP, Grade-Level 
Differences 



Number of LEAs 47 


Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 
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Exhibit 7C.14: Regression Results for Unsigned Claiming Error Rate in NLSP, Grade-Level 
Differences 



Coef. 

SE 

Sig. 

Treatment 

2.73 

2.66 


Treatment * Middle school 

-1.29 

4.02 


Treatment * High school 

2.11 

2.94 


Urban school 

-0.40 

1.79 


Charter schools 

-5.08 

3.02 

* 

Number of schools in LEA 

-6.92 

6.01 


% FRP students 

0.92 

2.82 


% Black students 

-8.44 

6.81 


% Hispanic/Latino students 

0.11 

0.11 


Middle school grades sampled 

1.92 

2.02 


High school grades sampled 

-0.85 

0.78 


Illinois 

2.90 

2.06 


Kentucky 

5.85 

2.84 

** 

Michigan 

7.03 

4.39 


New York 

7.19 

3.47 

** 

Ohio 

3.56 

2.99 


Constant 

1.66 

3.11 


Number of observations 

137 



Number of LEAs 

47 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 


Exhibit 7C.15: Regression Results for Total Signed Claiming Error Rate, Grade-Level 
Differences 



Number of LEAs 47 


Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 
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Exhibit 7C.16: Regression Results for Total Unsigned Claiming Error Rate, Grade-Level 
Differences 



Coef. 

SE 

Sig. 

Treatment 

2.79 

2.60 


Treatment * Middle school 

-1.25 

3.87 


Treatment * High school 

2.00 

2.78 


Urban school 

-0.56 

1.67 


Charter schools 

-4.92 

2.91 

* 

Number of schools in LEA 

-6.54 

5.86 


% FRP students 

0.85 

2.74 


% Black students 

-8.27 

6.65 


% Hispanic/Latino students 

0.11 

0.10 


Middle school grades sampled 

1.75 

1.83 


High school grades sampled 

-0.83 

0.76 


Illinois 

2.86 

2.02 


Kentucky 

5.74 

2.69 

** 

Michigan 

7.00 

4.28 


New York 

7.00 

3.36 

** 

Ohio 

3.57 

2.91 


Constant 

1.57 

3.01 


Number of observations 

137 



Number of LEAs 

47 




Reference categories are Elementary for grades sampled and West Virginia for state. 

***p<0.01; **p<0.05; * p<0.10 

Source: Meal Counting and Claiming Data Collection 
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Appendix 8A: NSLP and SBP Meal Standards 


Traditional Food-Based Menu Planning Approach-Meal Pattern for Breakfasts 


Food Components and Food Items 

Grades K-12 

MILK (fluid) (as a beverage, on cereal or both) 

8 fluid ounces 

JUICE/FRUIT/VEGETABLE: Fruit and/or vegetable; or full-strength fruit juice 
or vegetable juice 

cup 

SELECT ONE SERVING FROM EACH OF THE FOLLOWING 

COMPONENTS, TWO FROM ONE COMPONENT, ORAN EQUIVALENT 
COMBINATION: 


GRAINS/BREADS: 

Whole-grain or enriched bread 

Whole-grain or enriched biscuit, roll, muffin, etc. 

Whole-grain, enriched or fortified cereal 

1 slice 

1 serving 
% cup or 1 ounce 

MEA T OR MEA T AL TERN A TES: 

Meat/poultry or fish 

Alternative protein products 1 

Cheese 

Large egg 

Peanut butter or other nut or seed butters 

Cooked or dry beans and peas 

Nuts and/or seeds (as listed in program guidance) 2 

Yogurt, plain or flavored, unsweetened or sweetened 

1 ounce 

1 ounce 

1 ounce 

y 2 

2 Tablespoons 

4 Tablespoons 

1 ounce 

4 ounces or 'A cup 


Must meet the requirements in appendix A of 7 CFR 220. 

2 No more than 1 ounce of nuts and/or seeds may be served in any one breakfast. 

The Traditional Food-Based Menu Planning Approach is designed to meet nutritional standards set 
forth in program regulations. 
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Final Rule Nutrition Standards in the National School Lunch and School Breakfast Programs - Jan. 2012 



Breakfast Meal Pattern 

Lunch Meal Pattern 


Grades K-5 4 

Grades 6-8 a 

Grades 9-1 2 a 

Grades K-5 

Grades 6-8 

Grades 9-12 

Meal Pattern 

Amount of Food" Per Week (Minimum Per Day) 

Fruits (cups)' ,d 

5(1) ' 

5(1)' 

5(1)' 

214 (14) 

214 (54) 

5(1) 

Vegetables (cups)' 4 

0 

0 

0 

3% (%) 

3% (%) 

5(1) 

Dark green * 

0 

0 

0 

54 

14 

54 

Red/Orange 1 

0 

0 

0 

% 

% 

114 

Beans/Peas 
(Legumes) f 

0 

0 

0 

14 

14 

54 

Starchy' 

0 

0 

0 

14 

14 

54 

Other ■ g 

0 

0 

0 

54 

14 

% 

Additional Veg to 

Reach Total h 

0 

0 

0 

1 

1 

154 

Grains (oz eq) 1 

7-10 (1) J 

8-10 (1) J 

9-10 (1) J 

8-9(1) 

8-10(1) 

10-12(2) 

Meats/Meat Alternates 
(oz eq) 

o k 

o k 

o k 

8-10(1) 

9-10(1) 

10-12(2) 

Fluid milk (cups) 1 

50) 

50) 

50) 

50) 

50) 

50) 

Other Specifications: Daily Amount Based on the Average for a 5-Day Week 

Min-max calories 
(kcal) m,n ’° 

350-500 

400-550 

450-600 

550-650 

600-700 

750-850 

Saturated fat 

(% of total calories)"’ 0 

<10 

<10 

<10 

<10 

<10 

<10 

Sodium (mg)"’ p 

<430 

<470 

<500 

<640 

<710 

<740 

Trans fat"’ 0 

Nutrition label or manufacturer specifications must indicate zero grams of trans fat per serving. 


“In the SBP, the above age-grade groups are required beginning July 1, 2013 (SY 2013-14). In SY 2012-2013 only, schools may 
continue to use the meal pattern for grades K-12 (see § 220.23). 

b Food items included in each food group and subgroup and amount equivalents. Minimum creditable serving is 14 cup. 
c One quarter-cup of dried fruit counts as 14 cup of fruit; 1 cup of leafy greens counts as ‘A cup of vegetables. No more than half 
ofthe fruit or vegetable offerings may be in the form of juice. All juice must be 100% full-strength. 

d For breakfast, vegetables may be substituted for fruits, but the first two cups per week of any such substitution must be from the 
dark green, red/orange, beans and peas (legumes) or “Other vegetables” subgroups as defined in §210.10(c)(2)(iii). 

“The fruit quantity requirement for the SBP (5 cups/week and a minimum of 1 cup/day) is effective July 1, 2014 (SY 2014- 
2015). 

f Larger amounts of these vegetables maybe served. 

8 This category consists of “Other vegetables” as defined in §210.10(c)(2)(iii)(E). For the purposes ofthe NSLP, “Other 
vegetables” requirement may be met with any additional amounts from the dark green, red/orange, and beans/peas (legumes) 
vegetable subgroups as definedin §210.10(c)(2)(iii). 

h Any vegetable subgroup may be offered to meet the total weekly vegetable requirement. 

‘At least half ofthe grains offered must be whole grain-rich in the NSLP beginning July 1, 2012 (SY 2012-2013), and in the SBP 
beginning July 1, 2013 (SY 2013-2014). All grains must be whole grain-rich in both the NSLP and the SBP beginning July 1, 
2014 (SY 2014-15). 

J In the SBP, the grain ranges must be offered beginning July 1, 2013 (SY 2013-2014). 

'There is no separate meat/meat alternate component in the SBP. Beginning July 1, 2013 (SY 2013-2014), schools may 
substitute 1 oz. eq. of meat/meat alternate for 1 oz. eq. of grains after the minimum daily grains requirement is met. 

'Fluid milk must be low-fat (1 percent milk fat or less, unflavored) or fat-free (unflavored or flavored). 

“The average daily amount of calories for a 5-day school week must be within the range (at least the minimum and no more than 
the maximum values). 

“Discretionary sources of calories (solid fats and added sugars) may be added to the meal pattern if within the specifications for 
calories, saturated fat, trans fat, and sodium. Foods of minimal nutritional value and fluid milk with fat content greater than 1 
percent milk fat are not allowed. 

°Inthe SBP, calories and trans fat specifications take effect beginning July 1, 2013 (SY 2013-2014). 
p Final sodium specifications are to be reached by SY 2022-2023 or July 1,2022. Intermediate sodium specifications are 
established for SY 2014-2015 and 2017-2018. See required intermediate specifications in § 210.10(f)(3) for lunches and § 
220.8(f)(3) for breakfast 
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Final Rule "Nutrition Standards in the National School Lunch and School Breakfast Programs (1/26/12) 


Sodium Reduction: Timeline & Amount 


Baseline: 

Average 

Target 1: 

Target 2: 

Final Target: 

Age/Grade 

Current 

July 1, 2014 

July 1, 2017 

July 1, 2022 

Group 

Sodium Levels 





As Offered 1 

SY 2014-2015 

SY 2017 2018 

SY 2022-2023 


(mg) 

(mg) 

(mg) 

(mg) 

School Breakfast 

Program 

K-5 

573 

(elementary) 

<540 

<485 

<430 

6-8 

629 

(middle) 

<600 

<535 

<470 

9-12 

686 

(high) 

<640 

<570 

<500 

National School Lunch Program 

K-5 

1,377 

(elementary) 

< 1,230 

<935 

<640 

6-8 

1,520 

(middle) 

< 1,360 

< 1,035 

<710 

9-12 

1,588 

_(high)_ 

< 1,420 

< 1,080 

<740 


'SNDA-III 
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Appendix 8B: Menu Survey Processing for Meal Quality and Menu 
Variety Assessment 


This appendix describes the methods used to assess the nutrient content and meal component 
composition of NSLP and SBP meals offered, over a week, based on cafeteria managers’ responses to 
the menu surveys. This appendix also includes a discussion of the methods used to determine 
whether meals selected by students during the meal and cashier observations were reimbursable. 

8B.1 Review and Processing of Completed Menu Surveys 

Completed menu surveys were logged in to a tracking database upon receipt, and checked to assess 
whether all expected materials had been returned. Study staff followed up with cafeteria managers 
and in some cases LEA foodservice directors, by telephone or email to retrieve missing materials. 
Trained nutrition coders completed all data processing (e.g., cleaning and coding) and data entry into 
the nutrient analysis system. 1 

Each completed menu survey booklet was systematically reviewed, along with recipe information and 
production records, using fonn-specific procedures to (1) determine whether any key information was 
missing or needed clarification, and (2) cross-checked with school production records, other menu 
surveys from the same LEA, and with the meal observation booklets for details. If necessary, coders 
followed up with cafeteria managers or foodservice directors to obtain missing details or clarification 
for each menu survey before entering menu survey data into the nutrient analysis system. 

8B.2 Entry of Menu Survey Data into the Nutrient Analysis System 

Nutrition coders were trained to use the NUTR1K1DS menu planning software by Heartland School 
Solutions. The software is designed according to USDA specifications for the analysis of school 
meals. It includes the USDA Child Nutrition Database Release CN16 and USDA recipes for school 
foodservice. The software also includes an ingredient database of brand-specific foods that are 
commonly used in school foodservice operations. Due to software limitations and issues with 
multiple user interfaces, coders each used a separate database for entering school meal information on 
their assigned set of LEA menus. For each set of menus, the NUTRIKIDS software system generated 
nutrient and meal component output files, which were combined outside of the NUTRIKIDS 
program. 

A unique menu file was created for each school’s breakfast and lunch menus using the grade 
groupings (K-5, 6-8 and 9-12) as appropriate for the three schools in each LEA. The software 
requires every item on the menu, including single foods, be entered as a recipe. Coders created all 
recipes for a school before constructing the menu files. For consistency and efficiency, default recipes 
with appropriate food component crediting assigned were created by the trainer for commonly used 
school foods such as milk, common fruits and vegetables. Coders created new recipes or copied and 
modified default or existing USDA recipes included in the software for every food item listed on the 


The coders, primarily undergraduate nutrition students enrolled in local universities, were trained by an 
external expert with experience training school foodservice managers to use nutrient analysis software. 
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Reimbursable Food Form of the menu survey. Coders entered all recipes for one school, starting with 
the elementary school for each LEA, constructed the menu and then repeated the process for the 
middle, and then the high school. If ingredients were not in the CN or NUTRIKIDS brand name 
database, they were added to a list of missing Commercially Prepared Foods. Lead coders were 
responsible for finding the nutrient information online for the products on the list using information 
on manufacturer, brand name and product code provided in the menu survey document. Once 
obtained, the relevant information was entered into the database and communicated to the entire 
coding team, so that menu entry could be completed. 

8B.3 Meal Component and Subgroup Calculations 

During recipe creation, all foods that were not USDA recipes required the coder to assign food-based 
component and subgroup crediting, where applicable. The expert consultant trained coders on 
procedures for conducting component calculations for single component items, and using resources, 
primarily the USDA Food Buying Guide for determining creditable amounts of each contributing 
food component and subgroup for multiple component recipes, Coders used existing component 
values provided with USDA recipes and component values that were provided on CN label 
information obtained online for processed foods reported. The NUTRIKIDS software includes a 
module that allows for the assignment of meal component and relevant subgroup values per serving, 
and quantifies the daily summary (in cup equivalents for fruits, vegetables and milk; and in ounce 
weights for meat/meat alternates, grains and grain-based desserts). While the component crediting 
values must be manually entered for each recipe, the software automatically standardizes the 
component values by rounding the information entered by the coder down to the creditable 
component contribution. For meat/meat alternates and grains (measured in ounces) ,the software 
rounds the value down to the nearest quarter ounce equivalent; for fruits, vegetables and milk 
components (measured in cups), the software rounds the value down to the nearest eighth (1/8) cup. 
These procedures are consistent with the crediting guidelines issued by FNS to program operators 
(USDA, 2013). 

The evaluation team programmed the calculations for the unweighted meal component assessment for 
each school’s week of lunch menus using the May 2012 certification tool and guidance provided by 
FNS to all State CN directors for measuring compliance with the new meal requirements. The 
methodology used to assess meal quality for this evaluation sought to replicate the certification tool 
developed by FNS. LEAs complete the certification tool and submit the information to their State 
agencies in order to obtain certification as qualifying for the additional reimbursement authorized by 
the HHFKA. The LEA selects the menus and enters each reimbursable meal choice offered for each 
grade grouping. For each meal choice, the LEA enters all required food components (any 
grain/meat/meat alternates in a main dish and/or side dish, and amount of milk). The tool sums 
quantity information for each relevant meal component. The State uses the minimum of each food 
component in all meals offered to determine daily and weekly meal component compliance. The 
NUTRIKIDS software produces weekly compliance worksheets that follow the USDA guidance, but 
it was not feasible to use these for constructing the analysis files, due to the large number of menus 
and the format of the output files. However, the evaluation team used a sample of NUTRIKIDS 
weekly compliance worksheets to check that team’s analysis programs produced the same results. 

The study approach used information from the planned menu, the Menu Survey and production 
records from each school to group together foods that were offered together to constitute the 
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reimbursable meal choices in each daily menu. A senior project nutritionist identified separate entrees 
and other menu items and specifically linked them to complete the essential first step in the meal 
component compliance analysis. The full component minimum was assigned to all reimbursable 
combinations—if the combinations appeared to be available for all students. Results for any meals 
that did not meet the daily meal component minimums were reviewed and verified by project staff to 
ensure that foods had been assigned proper credit, as well as to ensure that no other foods had been 
offered with the meal. However, coders and nutritionists had to reconstruct the foods that made up 
each meal choice, relying on the combination of data from the Menu Survey, production records, and 
any follow-up notes or clarifications provided by the LEA. In most cases, schools were given the 
benefit of the doubt, but if more than one source revealed that a meal choice had insufficient 
offerings, only the components that appeared to be offered with the meal choice were credited, 
resulting in the meal choice being identified as not having met one or more standards. In most cases, 
the meals that did not meet the meal component standards in our analysis were most often those 
offered as an alternate meal, and were low on creditable amounts of meat/meat alternate or were 
offered without a grain. The rules governing the certification tool use the minimum value of each 
food component across all reimbursable meal choices, often the alternate meal, regardless of the 
proportion of students that take the meal. Thus, if there was a tendency for LEAs to provide 
incomplete data for alternate meals, these missing data woidd have had a disproportionate effect on 
the results. 

The levels of meal components determined by the evaluation’s methods may differ from results of the 
certification tool for two reasons. First, LEAs select the weekly menu for that will be reviewed by the 
State Agency, whereas the analysis in this report is based on data provided for a target week selected 
by the evaluation team. Second, LEAs have complete information on the foods making up each meal 
choice, whereas the evaluation may not always have had this information. Thus, both methods have 
potential biases: the certification tool may introduce an upward bias on compliance measures due to 
the LEA’s discretion in selecting the menus for analysis, while the evaluation methods may have a 
downward bias due to missing information, perhaps amplified by the role of alternate meals. As 
discussed below, the quality control process included several steps to minimize errors in the coding of 
menus. Compared with the certification tool, the study’s approach provides a somewhat more 
conservative and more objective assessment of meal component compliance in school meals. The 
evaluation team used the same methods for treatment and comparison schools, and so it is reasonable 
to assume that the methods did not affect the impact estimates. 

8B.4 Self-Serve Salad and Theme Bars 

Special procedures, based on methods of the SNDA-II study (Fox et al., 2001), were used to create 
recipes for self-serve (e.g., salad) and made-to-order (e.g., pasta) bars. LEAs completed a form to 
record the foods offered on such bars but did not specify the composition of a serving from the bar. A 
nutrition coder created an “average” food bar recipe for foods reported to be offered in a self-serve 
type of food bar by any school. These recipes were created to include some of each item offered on 
the self-serve bar; component and sub-group quantities were calculated accordingly. The coder used a 
guide and reference materials created by the expert consultant using examples of Self-Serve and 
Made to Order bars from the menu surveys. The recipe construction used default portion sizes for 
foods offered based on those used for the SNDA-II study, and the recipes were updated for the new 
meal standards according to the grade-specific differences in minimum fruit and vegetable serving 
sizes. This same procedure was followed to create recipes for condiment and finishing bars. 
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8B.5 Coding the Meal Observation Booklets 

As discussed in Chapter 7, data collectors observed meals as taken by students and recorded by 
cashiers for the analysis of cashier error, using meal observation booklets. Coding the meal 
observation booklets for the one day of the target week when meal and cashier observations occurred 
as part of the review and coding of each school’s menu surveys, because coders needed information 
on creditable amounts of meal components offered as well as information on the appropriate Offer 
versus Serve guidelines (where applicable) to determine whether individual observations met the 
requirements for a reimbursable meal. For breakfasts, coders used the Offer versus Serve guidelines 
for food-based menu planning that applied to SBP standards in effect in SY 2012-13. For lunches, 
coders used the Offer versus Serve standards consistent with the new meal pattern guidelines for 
lunch in effect during SY 2012-13. 

8B.6 Quality Control Procedures 

Any questions from coders were addressed by a supervising nutritionist as they arose. Weekly 
meetings with senior evaluation staff and the nutrition consultant allowed the team to resolve 
questions and clarify decision rules; such rules were documented throughout the coding process to 
ensure that issues were addressed consistently. Coders completed data entry for an initial LEA; those 
data were reviewed thoroughly by the supervising nutritionist, and feedback was provided. At two 
points during the entry process, all coder-assigned databases of recipes were downloaded and 
manually reviewed by a senior nutritionist to check for out-of-range values for individual serving 
sizes. The range checks were done with calories, fat, saturated fat and sodium. Out-of-range values 
found or suspect were checked against the original recipe and ingredient. Coders were provided with 
the edited database lists to research and resolve each issue identified. This was done in conjunction 
with a quality control check of each menu survey, completed by the coders who were assigned to 
check the work of other coders. The supervising nutritionist resolved issues as they were raised. 

As a final cleaning step, an output file of all menus was reviewed by a senior project nutritionist to 
assess proper meal and subgroup component crediting. This was done as part of the step where 
reimbursable menu items were li nk ed for the meal component analysis. 
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Appendix 8C: Analysis of Impacts on Meeting Nutritional Standards 
and Meal Variety 


This appendix presents supplementary information for Chapter 8 about the CEP’s impacts on 
meeting nutritional standards and variety for meals offered in the NSLP and SBP. The appendix 
begins with information on the data and results of balance testing. Next, univariate statistics for 
outcome variables, including food component subgroups are presented. The appendix concludes with 
the complete regression results supporting the impact estimates and statistics on the percentage of 
schools that offer dessert. 


8C.1 Data and Balance Testing 

8C.1.1 Balance Testing at LEA Level (Same for NSLP and SBP) 

The analysis file for the meal quality and variety was constructed using data from the Menu Survey, 
completed by school cafeteria managers as part of Component 3 data collection for the impact study, 
as described in Chapter 2. Two comparison LEAs selected for this data collection were found 
ineligible to participate. Therefore, the sample was drawn from 52 LEAs (27 treatment and 25 
comparison). Exhibit 8C. 1 repeats the analysis presented in Exhibit 2 for this set of LEAs. The 
substantive conclusions from Chapter 2 are unaltered: treatment and comparison LEAs for the menu 
quality and variety analysis are balanced on all LEA-level observed covariates. 
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Exhibit 8C.1: LEA Characteristics for Meal Quality and Variety Analysis Sample 



T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade distribution. Differences in grade distributions were jointly tested using MANOVA. 

***=p<0.01, **=p<0.05, *=p<0.1 

Source: Common Core of School Data, State Administrative Data, Verification Summary Report Data 

8C.1.2 Balance Testing at School Level 

Data were collected from 156 schools in the 52 LEAs in Component 3 data collection for Menu 
Survey (for lunch and breakfast). Characteristics of these schools are shown in Exhibit 8C.2. 
Treatment schools had more students eligible for free and reduced price meals and a higher 
percentage of students who were Black. Treatment schools also tended to be more urban, Title 1 or 
charter schools. On all other characteristics, the two groups of schools were statistically 
indistinguishable. The characteristics which were unbalanced between the treatment and comparison 
schools were included as control variables in the impact estimation model for lunch. 
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Exhibit 8C.2: Meal Quality and Variety Sample, School Level Measures 



Percent students Hispanic/Latino 6.9 (10.0) 10.3 (17.7) 


a The grade span refers to the actual grade coverage of a school. Grade coverage can be irregular, especially 
among charter schools, which may cover K-12, K-8, or other grade spans, all of which are included in the “Other” 
category in the table. The analysis has categorized these irregular grade spans according to three separate 
sampled grades: elementary (K-5), middle (6-8), and high (9-12). The table presents both the original grade span 
and the sampled grade category. 

T-tests were used to test for differences between treatment and comparison means for all variables except for 
grade span and grades sampled. Differences in grade span and grades sampled were jointly tested using a chi- 
square test. 

***=p<0.01, **=p<0.05, *=p<0.1 
Source: Common Core of School Data 


8C.2 Univariate Statistics for Outcome Variables 

8C.2.1 Meal Quality 

Exhibits 8C.3-8C.4 show the unadjusted percentages of treatment and comparison schools meeting 
the grade range-specific meal pattern requirements for daily and weekly minimums for specific meal 
components, as well as nutrient specifications for calories, and saturated fat for lunches offered, 
respectively. A comparison with the future Target 1 sodium standard is also included. Exhibit 8C.5 
shows the unadjusted percentages of treatment and comparison schools meeting the appropriate meal 
component daily minimums and Target 1 for sodium for breakfasts offered. The tables include tests 
of significant differences between treatment and comparison schools. Unlike the regression results, 
these tests of unadjusted differences do not control for the observed differences in school 
characteristics identified in the balance tests. 
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Exhibit 8C.3: Unadjusted Percentages of Schools Meeting the Daily and Weekly Minimum for 
Food Components for Lunches Offered, by Treatment and Comparison Schools 





Meets at Least 4 out of 

Meets at Least 3 out of 


Meets All 5 Days of the 
Target Week 

the 5 Days of the Target 
Week 

the 5 Days of the Target 
Week 


Comparison 

Treatment 

Comparison 

Treatment 

Comparison 

Treatment 

FOOD COMPONENTS 

Meets fruit minimum 

84 

90.1 

92 

93.8 

92 

96.3 

Meets daily fruit minimum 

84 

90.1 

94.7 

95.1 

96 

98.8 

Meets weekly fruit minimum 

92 

96.3 

92 

96.3 

92 

96.3 

Meets vegetable minimum 

57.3 

69.1 

80 

79 

90.7 

86.4 

Meets daily veg. Minimum 

57.3 

69.1 

82.7 

81.5 

94.7 

90.1 

Meets weekly veq. Minimum 

92 

90.1 

92 

90.1 

92 

90.1 

Meets grains minimum 

36 

42 

44 

58* 

46.7 

59.3 

Meets daily grains minimum 

49.3 

45.7 

62.7 

70.4 

74.7 

76.5 

Meets weekly grains 
minimum 

48 

63* 

48 

63* 

48 

63* 

Meets meat/meat alternate 
minimum 

45.3 

53.1 

50.7 

56.8 

50.7 

60.5 

Meets daily meat/meat 
alternate minimum 

70.7 

72.8 

78.7 

84 

82.7 

90.1 

Meets weekly meat/meat 
alternate minimum 

50.7 

60.5 

50.7 

60.5 

50.7 

60.5 

Meets milk minimum 

100 

100 

100 

100 

100 

100 

Meets daily milk minimum 

100 

100 

100 

100 

100 

100 

Meets weekly milk minimum 

100 

100 

100 

100 

100 

100 

Meets all minimum 

12 

14.8 

20 

30.9 

24 

37 

SUBGROUP MINIMUMS 

Vegetable subgroup 







minimum 







Meets weekly dark green 
minimum 

94.7 

92.6 

94.7 

92.6 

94.7 

92.6 

Meets weekly red/orange 
minimum 

90.7 

88.9 

90.7 

88.9 

90.7 

88.9 

Meets weekly legume 
minimum 

78.7 

60.5** 

78.7 

60.5** 

78.7 

60.5** 

Meets weekly starch 
minimum 

97.3 

92.6 

97.3 

92.6 

97.3 

92.6 

Meets weekly other minimum 

96 

87.7* 

96 

87.7* 

96 

87.7* 

Grains subgroup 
requirement 

Meets weekly 50% whole 
grains requirement 

52 

61.7 

52 

61.7 

52 

61.7 

Other requirements 

Meets milk type requirement 

96 

97.5 

96 

97.5 

96 

97.5 

Meets 2 oz/wk grain-based 
dessert requirement 

94.7 

98.8 

94.7 

98.8 

94.7 

98.8 

Meets weekly juice 50% of 
total veq requirement 

100 

100 

100 

100 

100 

100 

Meets weekly juice 50% of 
total fruit requirement 

100 

100 

100 

100 

100 

100 

Number of schools 

75 

81 

75 

81 

75 

81 


Meeting minimum (the first row for each component) means that the food components meet any daily and weekly 
minimum requirement for the respective grade range. 


Meets milk type requirement implies that the school does not serve flavored 1% milk or flavored or unflavored 
2% milk. 


pg. 8C-4 1 Appendix 8C 


Abt Associates 







































Community Eligibility Provision Evaluation: Final Report 


Meets 2 oz/wk grain-based dessert requirement implies that the school does not serve desserts during the week 
which total to more than 2 ounces of grain. 

The regression results for component subgroups and meeting other requirements for lunch shown in Exhibit 8C.3 
are not presented but are available if required. 

***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Exhibit 8C.4: Unadjusted Percentages of Schools Meeting the Daily and Weekly Minimum for 
Nutrient Specifications for Lunches Offered, by Treatment and Comparison Schools 





Within 5% of Meeting the 

Within 10% of Meeting 


Meets Requirements 

Requirements 

the Requirements 

Nutrients 

Comparison 

Treatment 

Comparison 

Treatment 

Comparison 

Treatment 

Meets calorie range 
specification 

44.0 

37.0 

68.0 

60.5 

86.7 

75.3* 

Below calorie range 
specification 

32.0 

26.0 

22.7 

18.5 

10.7 

12.3 

Above calorie range 
specification 

24.0 

37.0* 

9.3 

21.0** 

2.7 

12.3** 

Meets saturated fat 
specification 

89.3 

90.1 

94.7 

97.6 

98.7 

98.8 

Meets all nutrient 







specifications (calories 
and saturated fat) 

37.3 

34.6 

62.7 

59.8 

85.3 

74.1* 

Meets sodium target 1 
specifications 

61.3 

49.4 

70.7 

63.4 

80 

71.6 

Number of schools 

75 

81 

75 

81 

75 

81 


Meeting a specification means that a school met the daily requirements, on average, based on the weekly menu 
for the school’s grade range. For the column presenting the percentage within 5% of meeting the requirements, 
“Below Calorie Range Specification” means below 95% of the specification, and “Above Calorie Range 
Specification” means above 105% of the specification. For the columns presenting percentages within 10% of 
meeting the requirements, “Below Calorie Range Specification” means below 90% of the specification, and 
“Above Calorie Range Specification” means above 110% of the specification. 

The Target 1 sodium standard was used in the analysis for comparison purposes and will be in effect in 
SY 2014-15. See Appendix 8A. 

***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibit 8C.5: Unadjusted Percentages of Schools Meeting the Daily Minimum for Food 
Components and the Target 1 Sodium Specification for Breakfasts Offered, by Treatment and 
Comparison Schools 



Breakfast 

Comparison 

Treatment 

Food components 

Meets vegetable/fruit minimum 

100% 

97.5% 

Meets meat/grains minimum 

98.7 

93.8 

Meets milk minimum 

100.0 

100.0 

Meets milk type requirement 

97.3 

97.5 

Meets all components minimum (veg/fruit, meat/grain, milk) 

98.7 

91.4“ 

Nutrients 

Meets future sodium target 1 

61.3% 

59.3% 

Within 5% of future sodium target 1 

68 

66.7 

Within 10% of future sodium target 1 

76 

74.1 

Number of schools 

75 

81 


Meeting meal component requirements implies that the food components meet the daily minimum meal pattern 
requirement for grades K-12. 

Meets milk type requirement implies that the school does not serve flavored 1% milk or flavored or unflavored 
2% milk. 

The Target 1 sodium standard was used in the analysis for comparison purposes and will be in effect in 
SY 2014-15. See Appendix 8A. 

***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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8C.2.2 Menu Choice and Variety 

Exhibit 8C.6 and 8C.7 show the number of choices offered for each specific meal component 
category for lunches and breakfasts offered, respectively. The number of choices is an average of the 
daily number of choices offered over the period of a week during which data was collected. 

Exhibit 8C.6: Number of Choices Offered for Lunches in Participating and Non-Participating 
Schools 


Meal Component Choices 

Comparison 
Mean (SD) 

Lunch 

Treatment 

Mean (SD) 

Number of fruits/100% juices 

2.58 

1.93 

2.58 

1.42 

Number of vegetables 

2.32 

0.98 

2.48 

1.55 

Number of separate grains/breads 

0.52 

0.43 

0.68 

0.83 

Number of types of milks 

2.96 

0.67 

2.83 

0.71 

Number of entrees 

3.72 

2.86 

3.39 

2.61 

Number of desserts 

0.07 

0.16 

0.15** 

0.21 

Number of schools 


75 

81 



Entrees for lunches offered include meat/meat alternates and combination entrees composed of a meat/meat 
alternate and at least one other meal component (grain or vegetable). Counts of separate grains/breads that 
were not part of a multi-component item were computed separately. 

***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 

Exhibit 8C.7: Number of Choices Offered for Breakfasts in Participating and Non-Participating 
Schools 



Combination entrees for breakfasts include only entrees composed of a meat/meat alternate and at least one 
other meal component (typically a grain). Counts of meat/meat alternates and grains that were not part of a multi- 
component item were computed separately. 

***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibits 8C.8 and 8C.9 show the distribution of schools by the number of choices offered for each 
specific meal component category for lunches and breakfasts offered, respectively. 


Exhibit 8C.8: Percentage of Schools for Average Daily Number of Choices for Lunches Offered 



Entrees for lunches offered include meat/meat alternates and combination entrees composed of a meat/meat 
alternate and at least one other meal component (grain or vegetable). Counts of separate grains/breads that 
were not part of a combination entree were computed separately 


Source: Menu Survey Data 
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Exhibit 8C.9: Percentage of Schools for Average Daily Number of Choices Offered for 
Breakfasts 


Percentage of Schools with Average 
Daily Number of Choices for Breakfasts 
Offered 

Comparison Treatment 


Number of Fruits/Vegetables/100% juices (not included in an 
entree) 



Combination entrees for breakfasts include only entrees composed of a meat/meat alternate and at least one 
other meal component (typically a grain). Counts of meat/meat alternates and grains that were not part of a 
combination entree were computed separately 

Source: Menu Survey Data 


Abt Associates 


Appendix 8C | pg. 8C-9 







































Community Eligibility Provision Evaluation: Final Report 


8C.3 Regression Results 

8C.3.1 Meal Quality 

Thi s section begins with regression results for meal quality of lunches offered. Exhibits 8C.10-8C.14 
present the complete set of regression results for meeting food component requirements for lunches 
offered. The regression results for meeting the milk requirement are omitted, since all the schools 
meet the requirement. Exhibits 8C. 15 - 8C. 18 present the complete set of regression results for 
meeting nutrient specification standards for lunches offered. The remaining exhibits in this section 
provide regression results for meal quality of breakfasts offered. Exhibits 8C.19 - 8C.21 present the 
complete set of regression results for meeting meal pattern requirements for breakfasts offered. The 
regression results for meeting the milk type requirement for breakfast are omitted since all the schools 
meet the requirement. Exhibit 8C.22 presents the regression results for meeting nutrient specifications 
for the future sodium Target 1 for breakfasts offered. The vector of school characteristics used as 
control variables in all models included: 

• the school’s grade level (i.e., whether elementary, middle, or high), with elementary school as the 
reference category; 

• the State, with West Virginia as the reference State; and 

• LEA and school-level features where balance testing (detailed above) discerned a difference 
between treatment and comparison LEAs that was statistically significant at the 10 percent level, 
which included percent of students approved for free or reduced price meals, percent of students 
who were Black, and whether the school was in a city, or was a Title 1 or charter school. 
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Lunches Offered 

Food Component Requirements 


Exhibit 8C.10: Meeting Fruit Minimum for Lunches Offered: Regression Results 



Meeting a minimum implies meeting both the daily and weekly grade-range-specific minimums. 
Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Exhibit 8C.11: Meeting Vegetable Minimum for Lunches Offered: Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

0.218 

0.1 

** 

Middle school indicator 

0.151 

0.058 

** 

High school indicator 

0.065 

0.088 


Students free/reduced price (%) 

-0.36 

0.31 


Urban LEA 

0.017 

0.134 


Title 1 school 

-0.306 

0.141 

** 

Charter school 

-0.483 

0.203 

** 

Percent students Black 

-0.445 

0.161 

*** 

IL state Indicator 

0.211 

0.186 


KY state Indicator 

0.339 

0.162 

** 

Ml state Indicator 

0.297 

0.197 


NY state Indicator 

0.235 

0.22 


OH state Indicator 

0.218 

0.332 


Intercept 

0.822 

0.194 

*** 

Number of LEAs 

52 



Number of schools 

156 




Meeting a minimum implies meeting both the daily and weekly grade-range-specific minimums. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. . 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibit 8C.12: Meeting Grains Minimum for Lunches Offered: Regression Results 



Meeting a minimum implies meeting both the daily and weekly grade-range-specific minimums. 
Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Exhibit 8C.13: Meeting Meat/Meat Alternate Minimum for Lunches Offered: Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

0.104 

0.1 


Middle school indicator 

-0.171 

0.085 

** 

High school indicator 

-0.331 

0.106 

*** 

Students free/reduced price (%) 

0.267 

0.348 


Urban LEA 

-0.218 

0.132 


Title 1 school 

-0.295 

0.155 

* 

Charter school 

-0.005 

0.201 


Percent students Black 

0.166 

0.199 


IL state Indicator 

-0.071 

0.253 


KY state Indicator 

-0.257 

0.185 


Ml state indicator 

-0.345 

0.197 

* 

NY state indicator 

-0.395 

0.223 

* 

OH state indicator 

-0.114 

0.198 


Intercept 

0.901 

0.209 

*** 

Number of LEAs 

52 



Number of schools 

156 




Meeting a minimum implies meeting both the daily and weekly grade-range-specific minimums. 
Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibit 8C.14: Meeting All Meal Component Minimums for Lunches Offered: 
Results 

Regression 

Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

0.051 

0.058 


Middle school indicator 

0.013 

0.06 


High school indicator 

-0.124 

0.083 


Students free/reduced price (%) 

0.43 

0.16 

*** 

Urban LEA 

-0.075 

0.071 


Title 1 school 

-0.36 

0.111 

*** 

Charter school 

-0.065 

0.092 


Percent students Black 

-0.058 

0.109 


IL state Indicator 

0.061 

0.124 


KY state Indicator 

0.066 

0.106 


Ml state indicator 

-0.029 

0.075 


NY state indicator 

-0.01 

0.084 


OH state indicator 

0.141 

0.225 


Intercept 

0.164 

0.099 


Number of LEAs 

52 



Number of schools 

156 




Meeting all meal components implies that the school meets both the applicable daily and weekly minimums for 
the fruit, vegetable, grains, meat/meat alternate and milk components. . 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Nutrient Specifications 


Exhibit 8C.15: Meeting Calorie Range Specification for Lunches Offered: Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

-0.024 

0.075 


Middle school indicator 

0.023 

0.103 


High school indicator 

-0.182 

0.105 

* 

Students free/reduced price (%) 

-0.236 

0.254 


Urban LEA 

-0.159 

0.116 


Title 1 school 

-0.001 

0.142 


Charter SCHOOL 

0.24 

0.175 


Percent students Black 

-0.245 

0.2 


IL state indicator 

0.242 

0.143 

* 

KY state indicator 

0.234 

0.128 

* 

Ml state indicator 

0.183 

0.166 


NY state indicator 

0.303 

0.152 

* 

OH state indicator 

0.129 

0.206 


Intercept 

0.533 

0.156 

*** 

Number of LEAs 

52 



Number of schools 

156 




Meeting the calorie range specification implies meeting the average daily grade range-specific calorie range 
based on a week of menus. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibit 8C.16: Meeting Percent of Calories from Saturated Fat Specification for Lunches 
Offered: Regression Results 



Meeting the percent of calories from saturated fat specification implies meeting the average daily grade range- 
specific specification based on a week of menus. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibit 8C.17: Meeting Calorie Range and Percent of Calories from Saturated Fat 

Specifications for Lunches Offered: Regression Results 

Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

0.006 

0.081 


Middle school indicator 

-0.001 

0.101 


High school indicator 

-0.178 

0.099 

* 

Students free/reduced price (%) 

-0.298 

0.278 


Urban LEA 

-0.112 

0.119 


Title 1 school 

0.048 

0.123 


Charter school 

0.198 

0.184 


Percent students Black 

-0.173 

0.214 


IL state Indicator 

0.244 

0.148 


KY state indicator 

0.17 

0.133 


Ml state indicator 

0.163 

0.172 


NY state indicator 

0.255 

0.165 


OH state Indicator 

0.108 

0.227 


Intercept 

0.481 

0.174 

*** 

Number of LEAs 

52 



Number of schools 

156 




Meeting specifications implies meeting the average grade range-specific daily specifications based on a week of 
menus for both the calorie range and percent of calories from saturated fat. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibit 8C.18: Meeting Future Sodium Target 1 for Lunches Offered: Regression Results 



Meeting future sodium Target 1 implies meeting the. average daily grade range-specific specification based on a 
week of menus. 

The Target 1 sodium standard was used in the analysis for comparison purposes and will be in effect in 
SY 2014-15. See Appendix 8A. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Abt Associates 
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Breakfasts Offered 

Food Component Requirements 


Exhibit 8C.19: Meeting Vegetable/Fruit Minimum for Breakfasts Offered: Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

-0.018 

0.019 


Middle school indicator 

-0.027 

0.027 


High school indicator 

-0.04 

0.037 


Students free/reduced price (%) 

-0.152 

0.132 


Urban LEA 

-0.076 

0.065 


Title 1 school 

0.001 

0.012 


Charter school 

0.134 

0.111 


Percent students Black 

0.068 

0.066 


IL state indicator 

0.011 

0.02 


KY state indicator 

0.026 

0.024 


Ml state indicator 

-0.038 

0.036 


NY state indicator 

0.044 

0.044 


OH state indicator 

0.056 

0.057 


Intercept 

1.111 

0.097 

*** 

Number of LEAs 

52 



Number of schools 

156 




Meeting a Requirement implies meeting the daily minimum food component servings. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibit 8C.20: Meeting Meat/Grains Minimum for Breakfasts Offered: Regression Results 



Meeting the meat/grains requirement implies meeting the daily minimum food component servings for the 
combination of the meat/meat alternate and grain meal components. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Exhibit 8C.21: Meeting All Food Components Minimum for Breakfasts Offered: Regression 
Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

-0.082 

0.059 


Middle school indicator 

-0.043 

0.031 


High school indicator 

-0.006 

0.044 


Students free/reduced price (%) 

-0.043 

0.253 


Urban LEA 

-0.009 

0.116 


Title 1 school 

0.006 

0.055 


Charter school 

0.114 

0.144 


Percent students Black 

0.039 

0.106 


IL state indicator 

0.019 

0.048 


KY state indicator 

0.032 

0.043 


Ml state indicator 

-0.05 

0.068 


NY state indicator 

-0.144 

0.166 


OH state indicator 

0.035 

0.101 


Intercept 

1.049 

0.137 

*** 

Number of LEAs 

52 



Number of schools 

156 




Meeting all food components minimum implies that the school meets all the fruit/vegetable, meat/grains and milk 
daily serving minimums combined. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations.. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Nutrient Specifications 


Exhibit 8C.22: Meeting Future Sodium Target 1 for Breakfasts Offered: Regression Results 



Meeting the future sodium Target 1 implies meeting the average grade-range-specific daily specification based 
on a week of menus. 

The Target 1 sodium standard was used in the analysis for comparison purposes and will be in effect in 
SY 2014-15. See Appendix 8A. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 

8C.3.2 Menu Choice and Variety 

Exhibits 8C.23- 8C.33 present the complete set of regression results for the number of menu choices 
within each food component category, first for lunches and then for breakfasts offered. The vector of 
school characteristics used as control variables in all models included: 

• the school’s grade level (i.e., whether elementary, middle, or high), with elementary school as the 
reference category; 

• the State, with West Virginia as the reference state; and 

• LEA and school-level features where balance testing (detailed above) discerned a difference 
between treatment and comparison LEAs that was statistically significant at the 10 percent level, 
which included percent of students approved for free or reduced price meals, percent of students 
who were Black, and whether the school was in a city, or was a Title 1 or charter school. 
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Lunches Offered 

Exhibit 8C.23: Number of Fruits and 100% Fruit Juice Choices Offered for Lunches: 
Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

-0.216 

0.29 


Middle school indicator 

0.683 

0.19 

*** 

High school indicator 

1.536 

0.416 

*** 

Students free/reduced price (%) 

-0.52 

0.849 


Urban LEA 

0.984 

0.397 

** 

Title 1 school 

1.014 

0.448 

** 

Charter school 

-2.305 

0.56 

*** 

Percent students Black 

0.85 

0.684 


IL state indicator 

-1.369 

0.383 

*** 

KY state indicator 

0.645 

0.429 


Ml state indicator 

0.175 

0.508 


NY state indicator 

-0.378 

0.511 


OH state indicator 

0.21 

1.241 


Intercept 

1.159 

0.505 

** 

Number of LEAs 

52 



Number of schools 

156 




Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Exhibit 8C.24: Number of Vegetable Choices Offered for Lunches: Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

0.086 

0.263 


Middle school indicator 

0.382 

0.169 

** 

High school indicator 

0.719 

0.301 

** 

Students free/reduced price (%) 

1.151 

1.3 


Urban LEA 

0.609 

0.687 


Title 1 school 

0.167 

0.244 


Charter school 

-1.584 

1.172 


Percent students Black 

-0.35 

0.684 


IL state indicator 

-0.839 

0.239 

*** 

KY state indicator 

0.423 

0.322 


Ml state indicator 

0.417 

0.488 


NY state indicator 

-0.628 

0.494 


OH state indicator 

-0.688 

1.002 


Intercept 

1.13 

0.859 


Number of LEAs 

52 



Number of schools 

156 




Number of vegetable choices also includes 100% vegetable juice offerings. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Abt Associates 
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Exhibit 8C.25: Number of Separate Grain/Bread Choices Offered for Lunches: Regression 
Results 



Separate Grains are those that are not part of an entree. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Exhibit 8C.26: Number of Milk Choices Offered for Lunches: Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

-0.124 

0.154 


Middle school indicator 

0.064 

0.088 


High school indicator 

0.137 

0.121 


Students free/reduced price (%) 

0.178 

0.471 


Urban LEA 

0.035 

0.213 


Title 1 school 

0.026 

0.21 


Charter school 

-0.234 

0.324 


Percent students Black 

-0.31 

0.41 


IL state indicator 

-0.137 

0.227 


KY state indicator 

0.245 

0.243 


Ml state indicator 

-0.57 

0.281 

** 

NY state indicator 

0.2 

0.357 


OH state indicator 

-0.137 

0.311 


Intercept 

2.853 

0.319 

*** 

Number of LEAs 

52 



Number of schools 

156 




Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibit 8C.27: Number of Entree Choices Offered for Lunches: Regression Results 



Entrees for lunches offered include meat/meat alternates and combination entrees comprised of a meat/meat 
alternate and at least one other meal component (grain or vegetable). 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations.. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Exhibit 8C.28: Number of Dessert Choices Offered for Lunches: Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

0.087 

0.034 

** 

Middle school indicator 

0.037 

0.022 

* 

High school indicator 

0.093 

0.046 

** 

Students free/reduced price (%) 

-0.052 

0.105 


Urban LEA 

-0.021 

0.036 


Title 1 school 

0.027 

0.053 


Charter school 

-0.096 

0.052 

* 

Percent students Black 

-0.014 

0.066 


IL state indicator 

-0.127 

0.059 

** 

KY state indicator 

-0.079 

0.063 


Ml state indicator 

-0.106 

0.06 

* 

NY state indicator 

-0.086 

0.07 


OH state indicator 

-0.099 

0.067 


Intercept 

0.123 

0.069 

* 

Number of LEAs 

52 



Number of schools 

156 




Desserts include grain-based desserts, such as cakes, cookies, cobblers and pastries, as well as other desserts 
such as gelatin, pudding, sherbet or ice cream. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations.. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Abt Associates 
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Breakfasts Offered 

Exhibit 8C.29: Number of Fruit and Vegetable Choices Offered for Breakfasts: Regression 
Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

-0.528 

0.307 

* 

Middle school indicator 

0.543 

0.174 

*** 

High school indicator 

1.286 

0.276 

*** 

Students free/reduced price (%) 

-0.242 

0.921 


Urban LEA 

-0.454 

0.354 


Title 1 school 

0.426 

0.498 


Charter school 

0.18 

0.476 


Percent students Black 

-0.95 

0.551 

* 

IL state indicator 

-0.736 

0.479 


KY state indicator 

-1.261 

0.513 

** 

Ml state indicator 

-1.092 

0.575 

* 

NY state indicator 

-1.111 

0.596 

* 

OH state indicator 

-1.467 

0.472 

*** 

Intercept 

3.738 

0.558 

*** 

Number of LEAs 

52 



Number of schools 

156 




Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 

Exhibit 8C.30: Number of Grain Choices Separate from Combination Entrees Offered for 
Breakfasts: Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

-0.643 

0.673 


Middle school indicator 

1.618 

0.48 

*** 

High school indicator 

1.91 

0.633 

*** 

Students free/reduced price (%) 

-2.517 

1.878 


Urban LEA 

0.213 

0.726 


Title 1 school 

0.632 

0.926 


Charter school 

-3.141 

1.402 

** 

Percent students Black 

-0.017 

1.183 


IL state indicator 

-2.442 

0.924 

** 

KY state indicator 

0.234 

0.907 


Ml state indicator 

2.23 

1.487 


NY state indicator 

-1.985 

1.212 


OH state indicator 

-0.139 

0.972 


Intercept 

6.039 

1.092 

*** 

Number of LEAs 

52 



Number of schools 

156 




Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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Exhibit 8C.31: Number of Meat/Meat Alternate Choices Separate from Combination Entrees 
Offered for Breakfasts: Regression Results 

Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

-0.168 

0.149 


Middle school indicator 

0.299 

0.122 

** 

High school indicator 

0.421 

0.148 

*** 

Students free/reduced price (%) 

-0.112 

0.433 


Urban LEA 

0.342 

0.204 

* 

Title 1 school 

0.351 

0.203 

* 

Charter school 

-1.217 

0.28 

*** 

Percent students Black 

0.069 

0.381 


IL state indicator 

-0.681 

0.261 

** 

KY state indicator 

-0.686 

0.255 

*** 

Ml state indicator 

-0.426 

0.261 


NY state indicator 

-0.905 

0.28 

*** 

OH state indicator 

-0.845 

0.465 

* 

Intercept 

0.934 

0.296 

*** 

Number of LEAs 

52 



Number of schools 

156 




Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Exhibit 8C.32: Number of Milk Choices Offered for Breakfasts: Regression Results 


Variable 

Estimate 

Std. Err. 

Sig. 

Treatment 

-0.149 

0.17 


Middle school indicator 

0.127 

0.108 


High school indicator 

0.224 

0.14 


Students free/reduced price (%) 

-0.132 

0.408 


Urban LEA 

0.123 

0.226 


Title 1 school 

0.077 

0.225 


Charter school 

-0.194 

0.449 


Percent students Black 

0.347 

0.501 


IL state indicator 

-0.252 

0.308 


KY state indicator 

0.378 

0.243 


Ml state indicator 

-0.854 

0.348 

** 

NY state indicator 

-0.769 

0.393 

* 

OH state indicator 

-0.5 

0.538 


Intercept 

2.76 

0.269 

*** 

Number of LEAs 

52 



Number of schools 

156 




Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 


Abt Associates 


Appendix 8C | pg. 8C-25 








































Community Eligibility Provision Evaluation: Final Report 


Exhibit 8C.33: Number of Combination Entree Choices Offered for Breakfasts: Regression 
Results 



Combination entrees for breakfasts include only entrees comprised of a meat/meat alternate and at least one 
other meal component (typically a grain). Meat/Meat alternates and grains that were not part of a combination 
entree were computed separately. 

Reference category for grade is Elementary; Reference for state is West Virginia state indicator. 

Only school level variables that are not balanced were added due to the limited number of observations. 
***=p<0.01, **=p<0.05, *=p<0.1 
Source: Menu Survey Data 
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8C.4 Percentage of Schools that Offered Dessert 

Exhibit 8C.34-8C.37 show the unadjusted percentages of treatment and comparison schools that 
offered dessert at least once a week as a part of lunch. Desserts included grain-based desserts, such 
as cakes, cookies, cobblers, and pastries, as well as other desserts such as gelatin, pudding, sherbet or 
ice cream. 


Exhibit 8C.34: Percentage of Schools that Offered Dessert at Lunch at least Once a Week 



Comparison 

Treatment 

Desserts not offered 

74.7% 

51.9% 

Desserts offered at least once a week 

25.3% 

48.2% 

Total number of schools 

75 

81 


Source: Menu Survey Data 


Exhibit 8C.35: Percentage of Elementary Schools that Offered Dessert at Lunch at least Once 
a Week 



Comparison 

Treatment 

Desserts not offered 

76.9% 

60% 

Desserts offered at least once a week 

23.1% 

40% 

Total number of schools 

26 

30 


Source: Menu Survey Data 


Exhibit 8C.36: Percentage of Middle Schools that offered Dessert at Lunch at least Once a 
Week 



Comparison 

Treatment 

Desserts not offered 

70.8% 

52% 

Desserts offered at least once a week 

29.2% 

48% 

Total number of schools 

23 

25 

Source: Menu Survey Data 

Exhibit 8C.37: Percentage of High Schools that Offered Dessert at Lunch at least Once a Week 


Comparison 

Treatment 

Desserts not offered 

76% 

42.3% 

Desserts offered at least once a week 

24% 

57.7% 

Total number of schools 

26 

26 


Source: Menu Survey Data 


Abt Associates 
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